Insert using parquet option

[jharris126]

Feel free to close if this isn't the time or place for this sort of issue at this point in the project.

It would be cool to have the standard behavior insert the tuples from the arrow table as shown in example.cc but also have an optional parameter to instead write the arrow table to a parquet file and use hyper's COPY FROM PARQUET command to insert. For larger files, I assume this would allow a tradeoff where the user can sacrifice temporary intermediate storage of the parquet file but gain much more efficient data insert performance.

[WillAyd]

Nice idea. I have done this myself for S3 -> Redshift using the AWSwrangler library in Python. Do you know of other libraries that handle this natively? If so we can look at them and try to emulate what they are doing.

Alternately if the Hyper API ever allows columnar writing rather than row-wise writes we may be able to get the best of everything. I'm not sure if that's in the roadmap but would certainly help third party libraries like this

[jharris126]

I've only used arrow via python, but I assume these native parquet features are consistent throughout.

pq.write_table to drop to parquet
https://arrow.apache.org/docs/python/parquet.html#reading-and-writing-single-files

parquet_file.metadata to pass to a type mapper for hyper types
https://arrow.apache.org/docs/python/parquet.html#finer-grained-reading-and-writing