Error in cooler balance when high number of chromsomes

[lldelisle]

Hi,
I found a bug in version 0.8.10.
Here is a minimal test:

echo -e "chr1\t1000\tchr1\t1000" > valid.pairs.txt
for i in {1..10000}; do 
  echo -e "chr$i\t50000" >> fake.size
done
cooler csort -i tabix -c1 1 -c2 3 -p1 2 -p2 4 -o unique.csort.gz valid.pairs.txt fake.size 
cooler makebins -o bins.txt fake.size 20000
cooler cload tabix --assembly fake  -c2 3 -p2 4 bins.txt unique.csort.gz test.cool
cooler balance --cis-only test.cool

And I get:

Traceback (most recent call last):
  File "/home/ldelisle/miniconda2/envs/hicexplorer3.5.2/bin/cooler", line 10, in <module>
    sys.exit(cli())
  File "/home/ldelisle/.local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/ldelisle/.local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/ldelisle/.local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ldelisle/.local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ldelisle/.local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/ldelisle/miniconda2/envs/hicexplorer3.5.2/lib/python3.6/site-packages/cooler/cli/balance.py", line 257, in balance
    grp['bins'][name].attrs.update(stats)
  File "/home/ldelisle/miniconda2/envs/hicexplorer3.5.2/lib/python3.6/_collections_abc.py", line 841, in update
    self[key] = other[key]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/ldelisle/.local/lib/python3.6/site-packages/h5py/_hl/attrs.py", line 95, in __setitem__
    self.create(name, data=value, dtype=base.guess_dtype(value))
  File "/home/ldelisle/.local/lib/python3.6/site-packages/h5py/_hl/attrs.py", line 194, in create
    attr = h5a.create(self._id, self._e(tempname), htype, space)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5a.pyx", line 47, in h5py.h5a.create
RuntimeError: Unable to create attribute (object header message is too large)

However, it looks like the matrix is balanced (in my real case).

[nvictus]

Storing the weights succeeded. It was only storing extra metadata that failed: https://github.com/open2c/cooler/blob/master/cooler/balance.py#L464

Thanks for reporting this. It turns out HDF5 attributes can't be too big, which led to other issues with large numbers of contigs. We might have to switch to not using attributes to store this information.