Support of temporal column types

[sgoeschl]

Hi Alexander,

I'm looking at "DataFrame" to use it in of my pet projects

• How to support temporal column types, e.g. LocalDate or LocalDateTime? Want I want to do is to hide a parsed CSV or Excel sheet behind "DataFrame" but Excel has temporal column types
• Did you have a look at Commons CSV? Since you strive for minimal dependencies it might be not an option but I wanted to ask :-)

Siegfried

[nRo]

Hi Siegfried,

Thanks for your suggestions.
I plan to rework column types to allow easy definition of custom value types.
This would allow the definition of DateColumns.
Further, it would require the option to add hints for parsing (e.g. date format within a column).

A bridge for Commons CSV is a great idea to improve file format support.

[sgoeschl]

Hi Alexander,

drop me a note when there is something to integrate - seems that I some unexpected free time in summer due to COVID-19 :-)

Siegfried

[nRo]

Hi Siegfried,

Help is always greatly appreciated :)

I hope that I can work on custom value type support this weekend.
Further, I will try to create issues for other features and enhancements to better track progress.

[sgoeschl]

I'm ready to kick into action - https://issues.apache.org/jira/browse/FREEMARKER-144

[nRo]

So I started working on custom value type support in this branch: value-type-abstraction and create an issue #22
There are still some TODOs to support custom value types, described in #22

This approach would allow the creation of a Date ValueType that can support different formats for parsing and writing

[sgoeschl]

I got a proof of concept running integrating "nRo/Dataframe" into Apache FreeMarker CLI

[nRo]

Thats great!

Regarding temporal column types.
How would you expect date formats are handled per default?
Some kind of auto detection, or a default format that can be changed by giving hints when the data is read.

[sgoeschl]

You make the decision - both approaches sound good to me :-)

What I want to do:

• Transform Apache Commons CSV into a DataFrame and here I only handle Strings (partly implemented)
• Transform a Map into a DataFrame (partly implemented)
• Transform an Excel sheet (POI & getDateCellValue) to a DataFrame

So I guess I come from side of defining the DataFrame and then populating it

[nRo]

Okay, thanks for the feedbacks.
Unfortunately, I don't have much time at the moment.
But I will keep working on that over the next days.

[sgoeschl]

Mostly finished but not really happy with the code :-)

[sgoeschl]

Is the feature branch stable enough to do some preliminary integration?

[nRo]

The feature branch is pretty stable.
It only needs support for autodetection of custom types and some tests.

[nRo]

sorry for the delay, the custom value feature branch is now merged into master #22 .

I will now start looking into temporal column types.
I suppose that three different types would be required:

• Date
• DateTime
• Timespan

Each should be able to handle different formats.
Possible operations for these tables would be:

• convert them from one type into another
• operations for different time units (add minutes, hours, days,...)
• operations between columns (calculate difference,...)

Any other suggestions?

[sgoeschl]

Hi, little bit confused by "Timespan" - is this a time-only value, e.g. "13:42:23" or do you mean "3:24h"?

Regarding operations - little to no suggestion from my side - I would mostly filter / sort / query on temporal columns coming from CSV and Excel.