Skip to content

Commit

Permalink
Fix the null partition key filters
Browse files Browse the repository at this point in the history
  • Loading branch information
yingsu00 committed Mar 25, 2024
1 parent 1ed8075 commit e1ceaa2
Showing 1 changed file with 7 additions and 4 deletions.
11 changes: 7 additions & 4 deletions velox/connectors/hive/HiveConnectorUtil.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -581,16 +581,19 @@ bool testFilters(
for (const auto& child : scanSpec->children()) {
if (child->filter()) {
const auto& name = child->fieldName();
if (!rowType->containsChild(name)) {
// If missing column is partition key.
auto iter = partitionKey.find(name);
auto iter = partitionKey.find(name);
// The partition key columns are writen in the data file for
// IcebergTables, so we need to test both cases
if (!rowType->containsChild(name) || iter != partitionKey.end()) {
if (iter != partitionKey.end() && iter->second.has_value()) {
// This is a non-null partition key
return applyPartitionFilter(
(*partitionKeysHandle)[name]->dataType()->kind(),
iter->second.value(),
child->filter());
}
// Column is missing. Most likely due to schema evolution.
// Column is missing, most likely due to schema evolution. Or it's a
// partition key but the partition value is NULL.
if (child->filter()->isDeterministic() &&
!child->filter()->testNull()) {
return false;
Expand Down

0 comments on commit e1ceaa2

Please sign in to comment.