count+where query on Parquet format GHA Sample input is ~2.5x faster with "-dynamic"

[philrz]

The performance of the following query varies significantly based on whether the -dynamic flag is used.

$ time super -dynamic -c "
SELECT count() AS count
FROM 'ghasample.parquet'
WHERE actor.login='johnbieren';"
{count:2}

real    0m0.076s
user    0m0.071s
sys     0m0.028s

$ time super -c "
SELECT count() AS count
FROM 'ghasample.parquet'
WHERE actor.login='johnbieren';"
{count:2}

real    0m0.187s
user    0m0.590s
sys     0m0.408s

Details

Repro is with super commit 6731d1f.

$ super -version
Version: v0.3.0-244-g6731d1fd2

The input file is from s3://brim-sampledata/gha/ghasample.parquet. That input file is the frequently used "GitHub Archive Sample" data which started life as 22k JSON values each of unique type that was previously loaded into DuckDB as a table and then dumped out to a Parquet file.

The result above was taken on my Intel Macbook, but I've also reproduced it on an AWS EC2 m6idn.xlarge Linux instance.

It's been said that type checking with Parquet should be very cheap since there's only one schema per file, so the perf hit when invoking -dynamic is unexpcted.

Claude's take

I asked Claude to do some profiling and give its theory as to what's going on, so its findings can be reviewed in a Gist if that saves anyone time.