Skip to content

Parquet & CSV support #22

@marklit

Description

@marklit

I've captured telemetry from an X-Plane 12 flight. It allows you to record potentially 100s of sensors during any flight.

Image

I've recorded the video and audio with OBS and trimmed the resulting MP4 file to the beginning of the flight, when the telemetry began recording.

$ ffmpeg -i 2026-06-01\ 19-20-55.mp4 -ss 37 -vcodec copy -acodec copy xplane.mp4

I then trimmed all the white space out of the telemetry and produced a Parquet file. This can be rendered in QGIS and ArcGIS Pro without issue.

$ sed 's/ //g' Data.txt \
    | ~/duckdb -c "COPY(SELECT
                        geometry: ST_POINT(\"__lon,__deg\",
                                           \"__lat,__deg\"),
                        elevation: \"_elev,_surf\",
                        *
                        FROM read_csv('/dev/stdin'))
                   TO 'flight.parquet' (
                        FORMAT 'PARQUET',
                        CODEC 'ZSTD',
                        COMPRESSION_LEVEL 22,
                        ROW_GROUP_SIZE 15000);"

These are the fields in the resulting Parquet file.

.maxrows 100

SELECT   column_name,
         column_type[:30],
         null_percentage,
         approx_unique,
         min[:30],
         max[:30]
FROM     (SUMMARIZE
          FROM 'flight.parquet')
WHERE    approx_unique > 1
ORDER BY LOWER(column_name);
┌─────────────┬──────────────────┬─────────────────┬───────────────┬──────────────┬──────────────┐
│ column_name │ column_type[:30] │ null_percentage │ approx_unique │   min[:30]   │   max[:30]   │
│   varchar   │     varchar      │  decimal(9,2)   │     int64     │   varchar    │   varchar    │
├─────────────┼──────────────────┼─────────────────┼───────────────┼──────────────┼──────────────┤
│ +-ISA,FL340 │ DOUBLE           │            0.00 │             2 │ -5e-05       │ -4e-05       │
│ ____A,_ktas │ DOUBLE           │            0.00 │          3332 │ 649.24023    │ 652.36377    │
│ ____P,deg/s │ DOUBLE           │            0.00 │          5219 │ -270.89316   │ 71.33768     │
│ ____Q,__psf │ DOUBLE           │            0.00 │          3500 │ 0.0          │ 57.17165     │
│ ____Q,deg/s │ DOUBLE           │            0.00 │          5875 │ -34.82273    │ 88.75576     │
│ ____R,deg/s │ DOUBLE           │            0.00 │          5315 │ -62.98122    │ 292.21109    │
│ ____X,____m │ DOUBLE           │            0.00 │          3749 │ -21650.29297 │ -20742.93945 │
│ ____Y,____m │ DOUBLE           │            0.00 │          3650 │ 1122.82349   │ 1539.02991   │
│ ____Z,____m │ DOUBLE           │            0.00 │          3372 │ 25364.05469  │ 27633.01953  │
│ ___CG,ftMSL │ DOUBLE           │            0.00 │          4964 │ 3981.42822   │ 5333.42627   │
│ ___cg,ftref │ DOUBLE           │            0.00 │            33 │ 0.49438      │ 0.49473      │
│ ___vX,__m/s │ DOUBLE           │            0.00 │          4480 │ -59.07093    │ 55.38167     │
│ ___vY,__m/s │ DOUBLE           │            0.00 │          5297 │ -45.43877    │ 28.44555     │
│ ___vZ,__m/s │ DOUBLE           │            0.00 │          6758 │ -50.85358    │ 69.51609     │
│ __lat,__deg │ DOUBLE           │            0.00 │          1639 │ 36.25079     │ 36.27124     │
│ __lon,__deg │ DOUBLE           │            0.00 │           970 │ -113.24089   │ -113.23077   │
│ __mag,_comp │ DOUBLE           │            0.00 │          7452 │ 0.02393      │ 359.74353    │
│ __VIS,_tris │ DOUBLE           │            0.00 │           300 │ 790759.0     │ 1412658.0    │
│ __VVI,__fpm │ DOUBLE           │            0.00 │          3979 │ -8594.09277  │ 5339.36328   │
│ _baro,_inHg │ DOUBLE           │            0.00 │             4 │ 29.91963     │ 29.91966     │
│ _beta,__deg │ DOUBLE           │            0.00 │          5760 │ -82.43665    │ 55.58096     │
│ _dens,ratio │ DOUBLE           │            0.00 │          2125 │ 0.85299      │ 0.88856      │
│ _dist,___ft │ DOUBLE           │            0.00 │          5741 │ 0.22546      │ 19757.90625  │
│ _dist,___nm │ DOUBLE           │            0.00 │          3409 │ 4e-05        │ 3.25173      │
│ _elev,_surf │ DOUBLE           │            0.00 │           766 │ -1.0         │ 0.76814      │
│ _elev,stick │ DOUBLE           │            0.00 │           660 │ -1.0         │ 0.69139      │
│ _flap,handl │ DOUBLE           │            0.00 │            24 │ 0.0          │ 1.0          │
│ _flap,postn │ DOUBLE           │            0.00 │           156 │ 0.0          │ 1.0          │
│ _fuel,___lb │ DOUBLE           │            0.00 │           252 │ 255.30658    │ 263.28311    │
│ _gear,ftagl │ DOUBLE           │            0.00 │          3574 │ -2.06003     │ 1135.97974   │
│ _Mach,ratio │ DOUBLE           │            0.00 │          3526 │ 0.0          │ 0.21258      │
│ _Pdot,_d/ss │ DOUBLE           │            0.00 │          7273 │ -20958.74609 │ 2904.83691   │
│ _Qdot,_d/ss │ DOUBLE           │            0.00 │          6365 │ -482.21985   │ 6555.92334   │
│ _Rdot,_d/ss │ DOUBLE           │            0.00 │          4173 │ -4229.41895  │ 21985.10547  │
│ _real,_time │ DOUBLE           │            0.00 │          7054 │ 293.6131     │ 582.32385    │
│ _roll,__deg │ DOUBLE           │            0.00 │          5429 │ -23.01233    │ 104.28939    │
│ _slip,__deg │ DOUBLE           │            0.00 │          3616 │ -7.99998     │ 5.63809      │
│ _totl,_time │ DOUBLE           │            0.00 │          6045 │ 44.57135     │ 325.07449    │
│ _trim,_elev │ DOUBLE           │            0.00 │            15 │ 0.0          │ 0.83783      │
│ _Vind,__mph │ DOUBLE           │            0.00 │          4319 │ 0.0          │ 147.30852    │
│ _Vind,_keas │ DOUBLE           │            0.00 │          4754 │ 0.0          │ 129.89479    │
│ _Vind,_kias │ DOUBLE           │            0.00 │          2826 │ 0.0          │ 128.0076     │
│ _zulu,_time │ DOUBLE           │            0.00 │          5064 │ 2.0971       │ 2.17502      │
│ ailrn,_surf │ DOUBLE           │            0.00 │           508 │ -1.0         │ 1.0          │
│ ailrn,stick │ DOUBLE           │            0.00 │           575 │ -1.0         │ 1.0          │
│ alpha,__deg │ DOUBLE           │            0.00 │          5216 │ -165.85753   │ 148.90276    │
│ AMprs,_inHG │ DOUBLE           │            0.00 │          3680 │ 24.58694     │ 25.85914     │
│ AMtmp,_degC │ DOUBLE           │            0.00 │          3022 │ 4.43612      │ 7.1135       │
│ fuel1,__gal │ DOUBLE           │            0.00 │           215 │ 19.86845     │ 21.22732     │
│ fuel1,tankC │ DOUBLE           │            0.00 │           268 │ 5.87552      │ 9.38299      │
│ fuel2,tankC │ DOUBLE           │            0.00 │           295 │ 5.97579      │ 10.16156     │
│ Gload,_side │ DOUBLE           │            0.00 │          3963 │ -12.58655    │ 0.39115      │
│ Gload,axial │ DOUBLE           │            0.00 │          4648 │ -27.31591    │ 1.33597      │
│ Gload,norml │ DOUBLE           │            0.00 │          4879 │ -0.8722      │ 13.28547     │
│ gravi,_m/ss │ DOUBLE           │            0.00 │           133 │ 9.79329      │ 9.79455      │
│ hding,__mag │ DOUBLE           │            0.00 │          5355 │ 0.13707      │ 359.97922    │
│ hding,_true │ DOUBLE           │            0.00 │          6561 │ 0.00101      │ 359.97751    │
│ hobbs,_time │ DOUBLE           │            0.00 │          4617 │ 2.04211      │ 2.10215      │
│ hpath,_true │ DOUBLE           │            0.00 │          5691 │ 0.00829      │ 359.96078    │
│ lbrak,__add │ DOUBLE           │            0.00 │           212 │ 0.0          │ 1.0          │
│ LEtmp,_degC │ DOUBLE           │            0.00 │          3164 │ 4.55824      │ 9.11281      │
│ local,_time │ DOUBLE           │            0.00 │          7010 │ 19.0971      │ 19.17502     │
│ mavar,__deg │ DOUBLE           │            0.00 │           498 │ -10.38825    │ -10.38246    │
│ missn,_time │ DOUBLE           │            0.00 │          6045 │ 44.57135     │ 325.07449    │
│ nwhel,steer │ DOUBLE           │            0.00 │          3812 │ -5.89416     │ 23.50416     │
│ p-alt,ftMSL │ DOUBLE           │            0.00 │          2956 │ 3982.89551   │ 5334.90234   │
│ pitch,__deg │ DOUBLE           │            0.00 │          5903 │ -64.01899    │ 60.35745     │
│ rbrak,__add │ DOUBLE           │            0.00 │           256 │ 0.0          │ 1.0          │
│ ruddr,_surf │ DOUBLE           │            0.00 │            45 │ -0.06931     │ -0.0         │
│ ruddr,stick │ DOUBLE           │            0.00 │            45 │ -0.06931     │ -0.0         │
│ sbrak,handl │ DOUBLE           │            0.00 │             2 │ 0.0          │ 0.5          │
│ sbrak,postn │ DOUBLE           │            0.00 │           100 │ 0.0          │ 0.5          │
│ terrn,ftMSL │ DOUBLE           │            0.00 │          4287 │ 3919.44824   │ 4247.1543    │
│ total,___lb │ DOUBLE           │            0.00 │           220 │ 3291.4292    │ 3299.40576   │
│ toten,vario │ DOUBLE           │            0.00 │          4408 │ -2618.94922  │ 7235.77979   │
│ vpath,__deg │ DOUBLE           │            0.00 │          5919 │ -88.67152    │ 59.31114     │
│ Vtrue,_ktas │ DOUBLE           │            0.00 │          5238 │ 0.0          │ 138.51047    │
│ Vtrue,_ktgs │ DOUBLE           │            0.00 │          5195 │ 0.0          │ 137.34972    │
│ Vtrue,mphas │ DOUBLE           │            0.00 │          4951 │ 0.0          │ 159.395      │
│ Vtrue,mphgs │ DOUBLE           │            0.00 │          5230 │ 0.0          │ 158.05923    │
│ wbrak,__set │ DOUBLE           │            0.00 │             2 │ 0.0          │ 1.0          │
│ wbrak,_part │ DOUBLE           │            0.00 │             4 │ 0.0          │ 1.0          │
└─────────────┴──────────────────┴─────────────────┴───────────────┴──────────────┴──────────────┘

It appears GPStitch only accepts SRT and GPX files for these sorts of captures.

I couldn't find a way to include the ~100 different bits of telemetry but I was able to add an "extension" field to the GPX file and used it for the amount of fuel remaining.

import datetime as dt
import xml.etree.ElementTree as ET

import duckdb
from gpx import (
    Extensions,
    GPX,
    Latitude,
    Longitude,
    Metadata,
    Track,
    TrackSegment,
    Waypoint,
)


con = duckdb.connect(database=':memory:')
con.sql('INSTALL spatial; LOAD spatial')

# Register namespace prefix for cleaner XML output
MARKSBLOGG_TPX = "http://tech.marksblogg.com/xmlschemas/TrackPointExtension/v1"
ET.register_namespace("gpxtpx", MARKSBLOGG_TPX)

# Create extension element
tpx = ET.Element(f"{{{MARKSBLOGG_TPX}}}TrackPointExtension")

waypoints = []

for rec in con.execute("FROM 'flight.parquet' ORDER BY \"_zulu,_time\"")\
              .fetchdf()\
              .to_dict(orient="records"):
    hr = ET.SubElement(tpx, f"{{{MARKSBLOGG_TPX}}}lb")
    hr.text = str(rec['_fuel,___lb'])

    waypoints.append(
        Waypoint(lat=rec['__lat,__deg'],
                 lon=rec['__lon,__deg'],
                 ele=rec['elevation'],
                 extensions=Extensions(elements=[tpx]),
                 time=dt.datetime.now(dt.UTC) + dt.timedelta(seconds=rec['_zulu,_time'])))

track_segment = TrackSegment(trkpt=waypoints)
track = Track(name="Morning Run", trkseg=[track_segment])

# Create metadata
metadata = Metadata(
    name="My GPS Track",
    desc="A sample track",
    time=dt.datetime.now(dt.UTC),
)

GPX(
    creator="My Application",
    metadata=metadata,
    trk=[track],
).write_gpx("xplane.gpx")

The above turned the 735 KB of Parquet data into a 288 MB GPX file.

It would be nice if I could just use a Parquet file or CSV instead of a GPX or SRT file with GPStitch. There are Python bindings for DuckDB and it allows for 100s of GIS and other file formats to load with little more than calling FROM ST_READ(), FROM READ_PARQUET(), etc...

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions