Commit 623a6b1
committed
feat(dwio): Add DATE to TIMESTAMP widening coercion in DWIO readers
Reading a DATE column as TIMESTAMP previously failed at
schema-compatibility validation because DateType inherits from
IntegerType (kind() == INTEGER), and integer→timestamp is not a numeric
widening. DWIO readers now accept the coercion and emit Timestamp
(days × 86400 seconds, 0 nanos) at read time.
The widening is wired in three places. TypeUtils.cpp short-circuits
isCompatible() when the file type is DATE and the requested kind is
TIMESTAMP. SelectiveColumnReader gains convertDateToTimestampValues(),
dispatched from getIntValues() when the requested kind is TIMESTAMP and
the file type is DATE — it follows the same sourceRows iteration pattern
as upcastScalarValues but applies the days × 86400 conversion and stores
Timestamp structs. ParquetReader.cpp's convertType() for
ConvertedType::DATE now also accepts TIMESTAMP as a valid requested type,
in addition to DATE itself.
Test plan:
- velox/dwio/parquet/tests/reader/ParquetReaderTest.cpp:
- readDateColumnAsTimestamp
- readDateColumnAsTimestampWithNulls
- readRealColumnAsDouble, readRealColumnAsDoubleWithNulls — coverage for pre-existing FLOAT→DOUBLE widening, no production change in this commit.
- velox/dwio/dwrf/test/ReaderTest.cpp:
- readDateColumnAsTimestamp
- readDateColumnAsTimestampWithNulls1 parent 4513605 commit 623a6b1
8 files changed
Lines changed: 907 additions & 2 deletions
File tree
- velox/dwio
- common
- tests
- dwrf
- reader
- test
- parquet
- reader
- tests/reader
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
303 | 303 | | |
304 | 304 | | |
305 | 305 | | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
306 | 325 | | |
307 | 326 | | |
308 | 327 | | |
| |||
452 | 471 | | |
453 | 472 | | |
454 | 473 | | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
455 | 576 | | |
456 | 577 | | |
457 | 578 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
595 | 595 | | |
596 | 596 | | |
597 | 597 | | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
598 | 615 | | |
599 | 616 | | |
600 | 617 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
133 | | - | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
134 | 145 | | |
135 | 146 | | |
136 | 147 | | |
| |||
0 commit comments