[query] Memory error in VDS combiner after adding, then immidiately dropping a field.

[chrisvittal]

What happened?

In #14675 I replaced END with LEN in VDS. In doing so, I made sure that both fields were present so as to not break people's existing pipelines. I added a hidden _drop_end flag to read_vds in order to be able to (mostly in the combiner) not have the END field present. This lead to a strange code pattern:

https://github.com/chrisvittal/hail/blob/f39364c177e0b009589826b2c6b3cd36c3ec359d/hail/python/hail/vds/variant_dataset.py#L44-L46

When running the final VDS+VDS merge in test_combiner_run on the local backend, this failed with a memory error (in debug mode):

RuntimeException: invalid memory access: 140a68008/00000001: not in 140a58008/00010000

Applying this patch fixed test_combiner_run:

diff --git a/hail/python/hail/vds/variant_dataset.py b/hail/python/hail/vds/variant_dataset.py
index 0f851e7364..01be83a982 100644
--- a/hail/python/hail/vds/variant_dataset.py
+++ b/hail/python/hail/vds/variant_dataset.py
@@ -41,9 +41,14 @@ def read_vds(
         reference_data = hl.read_matrix_table(VariantDataset._reference_path(path), _intervals=intervals)
         variant_data = hl.read_matrix_table(VariantDataset._variants_path(path), _intervals=intervals)

-    reference_data = VariantDataset._add_len_end(reference_data)
+    # if LEN is missing, add it, _add_len is a no-op if LEN is already present
+    reference_data = VariantDataset._add_len(reference_data)
     if _drop_end:
-        reference_data = reference_data.drop('END')
+        if 'END' in reference_data.entry:
+            reference_data = reference_data.drop('END')
+    else:  # if END is missing, add it, _add_end is a no-op if END is already present
+        reference_data = VariantDataset._add_end(reference_data)
+
     vds = VariantDataset(reference_data, variant_data)
     if VariantDataset.ref_block_max_length_field not in vds.reference_data.globals:
         fs = hl.current_backend().fs

There was nothing in the IR that stood out when I examined it, but I will admit that I'm not the best at digging into it.

Version

https://github.com/chrisvittal/hail/tree/vds/repro-example

Relevant log output

E           hail.utils.java.FatalError: RuntimeException: invalid memory access: 140a68008/00000001: not in 140a58008/00010000
E
E           Java stack trace:
E           java.lang.RuntimeException: invalid memory access: 140a68008/00000001: not in 140a58008/00010000
E           	at is.hail.annotations.Memory.checkAddress(Memory.java:226)
E           	at is.hail.annotations.Memory.loadByte(Memory.java:130)
E           	at is.hail.annotations.Region$.loadByte(Region.scala:28)
E           	at is.hail.annotations.Region$.loadBit(Region.scala:86)
E           	at __C23148collect_distributed_array_matrix_native_writer.__m23333split_ToArray(Unknown Source)
E           	at __C23148collect_distributed_array_matrix_native_writer.apply_region478_486(Unknown Source)
E           	at __C23148collect_distributed_array_matrix_native_writer.apply_region16_503(Unknown Source)
E           	at __C23148collect_distributed_array_matrix_native_writer.apply_region14_529(Unknown Source)
E           	at __C23148collect_distributed_array_matrix_native_writer.apply(Unknown Source)
E           	at __C23148collect_distributed_array_matrix_native_writer.apply(Unknown Source)
E           	at is.hail.backend.BackendUtils.$anonfun$collectDArray$10(BackendUtils.scala:90)
E           	at is.hail.utils.package$.using(package.scala:673)
E           	at is.hail.annotations.RegionPool.scopedRegion(RegionPool.scala:166)
E           	at is.hail.backend.BackendUtils.$anonfun$collectDArray$9(BackendUtils.scala:89)
E           	at is.hail.backend.local.LocalBackend.$anonfun$parallelizeAndComputeWithIndex$4(LocalBackend.scala:150)
E           	at is.hail.utils.package$.using(package.scala:673)
E           	at is.hail.backend.local.LocalBackend.$anonfun$parallelizeAndComputeWithIndex$3(LocalBackend.scala:150)
E           	at is.hail.utils.package$.$anonfun$runAll$2(package.scala:1038)
E           	at is.hail.CancellingExecutorService.$anonfun$newTaskFor$2(package.scala:1090)
E           	at is.hail.CancellingExecutorService$CancellingTask.run(package.scala:1067)
E           	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
E           	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
E           	at is.hail.relocated.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
E           	at is.hail.CancellingExecutorService.execute(package.scala:1111)
E           	at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
E           	at is.hail.utils.package$.$anonfun$runAll$1(package.scala:1038)
E           	at scala.collection.Iterator.foreach(Iterator.scala:943)
E           	at scala.collection.Iterator.foreach$(Iterator.scala:943)
E           	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
E           	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
E           	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
E           	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
E           	at is.hail.utils.package$.runAll(package.scala:1038)
E           	at is.hail.utils.package$.$anonfun$runAllKeepFirstError$3(package.scala:1054)
E           	at is.hail.backend.local.LocalBackend.parallelizeAndComputeWithIndex(LocalBackend.scala:146)
E           	at is.hail.backend.BackendUtils.collectDArray(BackendUtils.scala:85)
E           	at __C22901Compiled.__m23019split_CollectDistributedArray_region3_27(Emit.scala)
E           	at __C22901Compiled.__m23019split_CollectDistributedArray(Emit.scala)
E           	at __C22901Compiled.__m22905begin_group_0_region15_103(Emit.scala)
E           	at __C22901Compiled.__m22905begin_group_0(Emit.scala)
E           	at __C22901Compiled.__m22903split_Block(Emit.scala)
E           	at __C22901Compiled.apply(Emit.scala)
E           	at is.hail.backend.local.LocalBackend.$anonfun$_jvmLowerAndExecute$3(LocalBackend.scala:186)
E           	at is.hail.backend.local.LocalBackend.$anonfun$_jvmLowerAndExecute$3$adapted(LocalBackend.scala:186)
E           	at is.hail.backend.ExecuteContext.$anonfun$scopedExecution$1(ExecuteContext.scala:144)
E           	at is.hail.utils.package$.using(package.scala:673)
E           	at is.hail.backend.ExecuteContext.scopedExecution(ExecuteContext.scala:144)
E           	at is.hail.backend.local.LocalBackend.$anonfun$_jvmLowerAndExecute$2(LocalBackend.scala:186)
E           	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
E           	at is.hail.backend.local.LocalBackend._jvmLowerAndExecute(LocalBackend.scala:186)
E           	at is.hail.backend.local.LocalBackend._execute(LocalBackend.scala:212)
E           	at is.hail.backend.local.LocalBackend.$anonfun$execute$2(LocalBackend.scala:277)
E           	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
E           	at is.hail.backend.local.LocalBackend.$anonfun$execute$1(LocalBackend.scala:272)
E           	at is.hail.backend.local.LocalBackend.$anonfun$execute$1$adapted(LocalBackend.scala:271)
E           	at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:78)
E           	at is.hail.utils.package$.using(package.scala:673)
E           	at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:78)
E           	at is.hail.utils.package$.using(package.scala:673)
E           	at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:13)
E           	at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:65)
E           	at is.hail.backend.local.LocalBackend.$anonfun$withExecuteContext$2(LocalBackend.scala:120)
E           	at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:55)
E           	at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:62)
E           	at is.hail.backend.local.LocalBackend.withExecuteContext(LocalBackend.scala:105)
E           	at is.hail.backend.local.LocalBackend.execute(LocalBackend.scala:271)
E           	at is.hail.backend.BackendHttpHandler.handle(BackendServer.scala:88)
E           	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
E           	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
E           	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
E           	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:848)
E           	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
E           	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:817)
E           	at jdk.httpserver/sun.net.httpserver.ServerImpl$DefaultExecutor.execute(ServerImpl.java:201)
E           	at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.handle(ServerImpl.java:560)
E           	at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:526)
E           	at java.base/java.lang.Thread.run(Thread.java:829)
E
E
E
E           Hail version: 0.2.132-f39364c177e0
E           Error summary: RuntimeException: invalid memory access: 140a68008/00000001: not in 140a58008/00010000