Visibility Without Reconstruction: AIID Public-Record Reconstructability Audit

This replication package accompanies Visibility Without Reconstruction: Evidence from a 100-Incident AIID Audit, a preliminary empirical memo based on a single-coder audit of 100 AIID incidents. Intercoder reliability testing and blind validation of source-mechanism coding are planned before formal publication. The audit evaluates whether AIID-linked public records contain the causal, technical, and institutional evidence needed for protected reconstructability. The memo should be read as a preliminary portfolio and methods artifact, not as a formally validated publication.

Data source

The audit uses the official AIID database snapshot dated 2026-05-18 11:21 AM: backup-20260518112157.tar.bz2.

Direct GraphQL access to the AIID endpoint returned 403 Forbidden: Invalid origin, so the official snapshot was used as the fixed dataset cutoff.

Suggested AIID citation: McGregor, S. (2021). “Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database.” Proceedings of the Thirty-Third Annual Conference on Innovative Applications of Artificial Intelligence.

Raw files used

The audit was generated from the following AIID snapshot files:

• incidents.csv
• reports.csv
• classifications_CSETv1.csv
• classifications_GMF.csv
• classifications_MIT.csv
• license.txt

Raw AIID snapshot files and full third-party report text are not included in this public package.

Repository structure

• 01_memo/ — memo PDF
• 02_methods/ — methods appendix, codebook, decision rules, sample criteria, source-mechanism rules
• 03_data/ — locked sample, final coded dataset, analyzed results, summary tables
• 04_code/ — scripts for analysis, figure generation, and secondary cross-tabulations
• 05_figures/ — generated figures and tables
• 06_validation/ — validation plan for intercoder reliability and blind source-mechanism review

Reproduce analysis outputs

The public package can regenerate the analyzed tables and figures from the included coded data. Earlier pipeline scripts for candidate-pool construction and replacement sampling are included for transparency, but they require local raw AIID snapshot files and manually completed screening files that are not included in this public package.

Create a virtual environment and install dependencies:

python3 -m venv .venv
./.venv/bin/python3 -m pip install -r requirements.txt

Regenerate summary tables:

./.venv/bin/python3 04_code/analyze_coding_results.py

Regenerate figures:

./.venv/bin/python3 04_code/make_figures.py

Generate the accepted-risk/source-mechanism cross-tabulation:

./.venv/bin/python3 04_code/accepted_risk_crosstab.py

Main outputs

Key memo files

• 01_memo/Visibility_Without_Reconstruction_Preliminary_Empirical_Memo_May_2026.pdf

Key data outputs

• 03_data/coding_full_v1.1.csv
• 03_data/coding_results_analyzed.csv
• 03_data/summary_overall.csv
• 03_data/summary_by_category.csv
• 03_data/summary_by_subfield.csv
• 03_data/summary_by_source_mechanism.csv
• 03_data/summary_accepted_risk_by_mechanism.csv

Key figures

• 05_figures/category_mean_scores.png
• 05_figures/score_distribution.png
• 05_figures/source_mechanism_scores.png
• 05_figures/mean_score_by_incident_class.png
• 05_figures/mean_score_by_incident_form.png
• 05_figures/mean_score_by_actor_record_split.png

Key scripts

• 04_code/analyze_coding_results.py — validates the coding file and generates analyzed data and summary tables.
• 04_code/make_figures.py — regenerates figures and supporting figure tables.
• 04_code/accepted_risk_crosstab.py — generates the accepted-risk/source-mechanism cross-tabulation used in the secondary-pattern discussion.