-
Notifications
You must be signed in to change notification settings - Fork 174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incorrect offsets for ListArray with invalid lists #2723
Comments
It looks like this isn't an issue with let field = Field::new("foo", DataType::FixedSizeList(Box::new(DataType::Int32), 3));
let flat_child = Int32Array::from(("foo", (0..9).collect::<Vec<i32>>()));
let raw_validity = vec![true, false, true];
let validity = Some(arrow2::bitmap::Bitmap::from(raw_validity.as_slice()));
let arr = FixedSizeListArray::new(field, flat_child.into_series(), validity);
println!("{:?}", arr.to_arrow());
// FixedSizeListArray[[0, 1, 2], None, [6, 7, 8]]
let list_dtype = DataType::List(Box::new(DataType::Int32));
let list_arr = arr.cast(&list_dtype)?;
let l = list_arr.list()?;
println!("{:?}", l.to_arrow());
// LargeListArray[[0, 1, 2], None, [3, 4, 5]] so it looks like the validity is not being applied correctly when downcasting to a list. |
Thanks for the quick reply. Correct, I misunderstood how offsets should be set for invalid values because I took inspiration from the problematic snippet in the casting logic. 🙏 |
@michaelvay I think #2729 should have fixed the issue -- I'll go ahead and close this but please do re-open the issue if it continues to be a problem after the next release! |
@jaychia @universalmind303 Indeed solved thanks!! |
Describe the bug
Getting a list from a
ListArray
that contains an invalid value beforehand returns wrong list value.I think the issue is that the size of the invalid List is not expressed properly in the offsets resulting this issue.
To Reproduce
Steps to reproduce the behavior:
Daft/src/daft-core/src/array/ops/get.rs
Daft version: version = "0.3.0-dev0"
I aligned with
main
on this commit 20ffed4The text was updated successfully, but these errors were encountered: