[DENG-11196] Port socorro import job from spark

[BenWu]

Description

This ports the spark job at https://github.com/mozilla/telemetry-airflow/blob/3bdbe160fdf469e7630b8b87962a9c1883512be4/jobs/socorro_import_crash_data.py to a python script that loads the socorro_crash_v2 table. Output should be the same.

Depends on GCS permissions https://github.com/mozilla/webservices-infra/pull/11742

Related Tickets & Documents

• DENG-11196

Reviewer, please follow this checklist

[github-actions]

This PR ports the Socorro crash-report import from the Spark/Dataproc + parquet2bigquery pipeline to an in-repo query.py that runs a single BigQuery load into a temp table followed by a transform.sql that rewraps arrays and writes the crash_date partition. It adds README.md, query.py, transform.sql, and updates metadata.yaml.

The overall approach is clean — deriving the load schema from schema.yaml at runtime to keep it in sync, and using a temp table + transform to reproduce the Spark array-wrapping encoding is a reasonable design. My main concern is column ordering in transform.sql: writing to an existing partitioned table matches columns by position, so the EXCEPT-and-append reordering will likely fail the partition write, and the --dry-run path won't surface it because it sets no destination. Details inline.

[github-actions]

issue: This * EXCEPT (...) + append pattern reorders the output columns relative to schema.yaml. additional_minidumps (pos 9), addons (pos 10), json_dump, and memory_report are removed from their positions in the middle of the schema and re-emitted at the very end, while everything after them shifts up. BigQuery matches query results to an existing partitioned table (table$YYYYMMDD) by ordinal position, not by name, and a partition-decorator write cannot alter the table schema. So at the position where the table expects additional_minidumps (RECORD) the query now supplies addons_checked (BOOLEAN), and the write to the partition will fail with a schema mismatch.

Note this won't be caught by --dry-run: the dry-run QueryJobConfig sets no destination, so it validates the query in isolation and never compares against the destination table's column order.

Emit the columns in schema.yaml order — either rewrap each array in place instead of EXCEPT-and-append, or wrap the whole thing in an outer SELECT that projects the columns in the table's order.

[BenWu]

I'm not sure if this is true because we have a lot of ETL with non-matching schema order, but I'll test if I can get access to the source data.

[github-actions]

nitpick: This says crash_date is (DATE, REQUIRED), but schema.yaml declares it mode: NULLABLE. Reconcile the two so the doc matches the schema (and, if the existing prod table has crash_date as REQUIRED, update schema.yaml to match to avoid a mode mismatch on schema deploy).

[scholtzan]

Integration report

• View full diff, if available (SQL + DAGs) (requires private-bigquery-etl access)
• Private CI run
• Bigeye monitoring dry-run: ✅ plan clean — view plan (requires private-bigquery-etl access)