test#1444
Closed
FelixYBW wants to merge 25 commits into
Closed
Conversation
PR not submitted to upstream yet Alchemy-item: (ID = 880) [OAP] Register merge extract companion agg functions without suffix commit 1/1 - 76137f3
Signed-off-by: Yuan <yuanzhou@apache.org> Alchemy-item: (ID = 878) [OAP] Support struct schema evolution matching by name commit 1/1 - 58e3d0104
Alchemy-item: (ID = 883) [OAP] [13620] Allow reading integers into smaller-range types commit 1/1 - 4cae2f5
… outer join Signed-off-by: Yuan <yuanzhou@apache.org> Alchemy-item: (ID = 882) [OAP] [11771] Fix smj result mismatch issue commit 1/1 - ada7dd2
… and anti join Address comments disable by default Alchemy-item: (ID = 884) [OAP] Build hash table while adding input rows for left semi and anti join commit 1/1 - b7c9034
Alchemy-item: (ID = 901) [OAP] [14722] Fix memory leak caused by asynchronous prefetch commit 1/1 - 075e7909f13e178e78dce75dd1486795e9e93bc4
…facebookincubator#15477)" This reverts commit 1895711. Alchemy-item: (ID = 892) Iceberg staging hub commit 2/15 - 83cfc5a9aa8de7276ab57cceae68a437363eadd2
…incubator#15443)" This reverts commit 51d4a94. Alchemy-item: (ID = 892) Iceberg staging hub commit 3/15 - 22c01d19d926b95a0200185f233d80d00f11f22b
The function toValues removes duplicated values from the vector and return them in a std::vector. It was used to build an InPredicate. It will be needed for building NOT IN filters for Iceberg equality delete read as well, therefore moving it from velox/functions/prestosql/InPred icate.cpp to velox/type/Filter.h. This commit also renames it to deDuplicateValues to make it easier to understand. Alchemy-item: (ID = 892) Iceberg staging hub commit 5/15 - bb3a4e5c58326a1b7d46a7cb4e4aeb544ade4e73
This commit introduces EqualityDeleteFileReader, which is used to read Iceberg splits with equality delete files. The equality delete files are read to construct domain filters or filter functions, which then would be evaluated in the base file readers. When there is only one equality delete field, and when that field is an Iceberg identifier field, i.e. non-floating point primitive types, the values would be converted to a list as a NOT IN domain filter, with the NULL treated separately. This domain filter would then be pushed to the ColumnReaders to filter our unwanted rows before they are read into Velox vectors. When the equality delete column is a nested column, e.g. a sub-column in a struct, or the key in a map, such column may not be in the base file ScanSpec. We need to add/remove these subfields to/from the SchemaWithId and ScanSpec recursively if they were not in the ScanSpec already. A test is also added for such case. If there are more than one equality delete field, or the field is not an Iceberg identifier field, the values would be converted to a typed expression in the conjunct of disconjunts form. This expression would be evaluated as the remaining filter function after the rows are read into the Velox vectors. Note that this only works for Presto now as the "neq" function is not registered by Spark. See https://github.com/ facebookincubator/issues/12667 Note that this commit only supports integral types. VARCHAR and VARBINARY need to be supported in future commits (see facebookincubator#12664). Co-authored-by: Naveen Kumar Mahadevuni <Naveen.Mahadevuni@ibm.com> # Conflicts: # velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp # Conflicts: # velox/dwio/common/ScanSpec.h # Conflicts: # velox/type/Filter.h Alchemy-item: (ID = 892) Iceberg staging hub commit 6/15 - ab03fe536f00a6a7b6ba57cdccc05491eb7e5ba5
# Conflicts: # velox/connectors/hive/HiveConnectorUtil.cpp Alchemy-item: (ID = 892) Iceberg staging hub commit 7/15 - a2df8cc88a68d0d8bf8eac72582816e0d6901a26
Co-authored-by: Chengcheng Jin <Chengcheng.Jin@ibm.com> Alchemy-item: (ID = 892) Iceberg staging hub commit 8/15 - 2f38991a394065355a416a5a506a56ed9cc386c2
Alchemy-item: (ID = 892) Iceberg staging hub commit 9/15 - 3c008cecd79cb3776a30420182dc47f2c7927e4d
Alchemy-item: (ID = 892) Iceberg staging hub commit 10/15 - 7a803385b6fafa8600078db60e278d0349139495
…finity. Alchemy-item: (ID = 892) Iceberg staging hub commit 11/15 - 4206a6230387190fdef5b4f0fc7d900a79d5293e
Alchemy-item: (ID = 892) Iceberg staging hub commit 12/15 - 963f94ec9648eb13cc9bac76dc7cb60eab73b6dd
Alchemy-item: (ID = 892) Iceberg staging hub commit 13/15 - d0d86ef207bbfb6a8b8f14c325993a9e1e0304ba
Alchemy-item: (ID = 892) Iceberg staging hub commit 14/15 - f7109aae47be212a6d093acd53404f68e781cdd1
Alchemy-item: (ID = 892) Iceberg staging hub commit 15/15 - a24bf4fadf982a5c90f0f4cbdb5b21e719aa0350
Signed-off-by: Yuan <yuanzhou@apache.org> Alchemy-item: (ID = 906) fix: Adding daily tests commit 1/2 - e2eb2c6
we can cache ccache on every build even on failure, since ibm/velox is always incremental build Alchemy-item: (ID = 906) fix: Adding daily tests commit 2/2 - 0899ddc
Signed-off-by: Yuan <yuanzhou@apache.org> Alchemy-item: (ID = 756) fix: Remove website folder to bypass the security issues commit 1/1 - f7b127b5b45ee4c94c7aeec726f3bd01f1cc75cd
FelixYBW
force-pushed
the
test1
branch
2 times, most recently
from
4f54358 to
5ea36b1
Compare
Signed-off-by: Yuan <yuanzhou@apache.org> Alchemy-item: (ID = 902) fix: Disable flaky test temporary commit 1/1 - 0bc5c50ea906911a666eda3403b6cd7f2e95a754 test2 delete cache test2 gh gh gh gh
prestodb-ci
force-pushed
the
main
branch
5 times, most recently
from
5c76e5d to
133475a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.