Update to htslib 1.10.2 (without musl)

[johanneskoester]

Original PR: #184

• Pull 1.10.2 submodule. Initial pass on i32->i64 type conversion, will eventually fix issue update to htslib 1.10 #183

• Fixes issue Would be helpful if the installation instructions included the required package names #109 while I'm at it, long hanging fruit fix

• Fix compression levels

• Going through https://github.com/samtools/htslib/blob/develop/README.large_positions.md to spot 64 bit reference position changes and sam_* function renaming. Only 4 errors left, but need to closely check types afterwards very carefully. Specially for the binary formats where this change does not apply and (sub)fields that do not necessarily require big representations

• Repoint submodule to upstream release tag

• Switch from rust-bio hosted htslib to upstream's samtools htslib.

• Try to amend submodule

• Handle submodules manually on TravisCI, since it is failing now: https://travis-ci.org/rust-bio/rust-htslib/builds/645772614

• Odd balls that would need review marked with XXX. New htslib seems to require libcurl

• Add curl-sys crate for hplugin_curl, aws s3 et al

• Add conditional libcurl compilation flags

• Be nice with cargo fmt [ci skip]

• Try installing libcurl meta-package instead of the openssl dependent one?: https://travis-ci.org/rust-bio/rust-htslib/builds/645790138#L1529

• Does not work with libcurl-dev metapackage: https://travis-ci.org/rust-bio/rust-htslib/jobs/645812050#L166, rollback and try with bionic ubuntu distro. It does compile locally on OSX

• Add matrix compile support for TravisCI, I would like to know why it compiles fine in OSX and fails on Linux now...

• PPA source no longer needed for ubuntu bionic? [ci skip]

• Bionic packages htslib with gnutls instead of openssl: https://packages.ubuntu.com/source/disco/htslib ... and it even comes with pre-shipped libcurl4-gnutls-dev, so no need to install it explicitly?: https://travis-ci.org/rust-bio/rust-htslib/jobs/645816116#L160

• sed is not gsed on OSX... not entirely sure this substitution is 100% needed

• Add libssl-dev to the mix for the openssl-sys: https://travis-ci.org/rust-bio/rust-htslib/jobs/645825022#L1000. Testsuite seems to run partially now: https://travis-ci.org/rust-bio/rust-htslib/jobs/645825022#L623

• Add build-essential in hopes to fix libcurl issue, incorporate when including hts-sys, don't use default features -- let the outer r… #185, fix one testsuite warning

• Add curl on default features for hts-sys

• Point to the right htslib upstream repo, thanks @dlaehnemann [skip ci]

• Appease OSX musl-cross toolchain requirements: https://travis-ci.org/rust-bio/rust-htslib/jobs/645907695#L1222

• TravisCI timesout installing brew deps: https://travis-ci.org/rust-bio/rust-htslib/jobs/647667984#L212 ... Not sure if travis_wait will work with homebrew addons?: https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received... trying

• Try workaround for https://twitter.com/braincode/status/1226103002550849542

• Do not update homebrew (time intensive), switch from travis_wait to verbose to keep the build output being generated, hopefully

• brew install --verbose is not verbose enough for travis, reverting travis_wait and bumping to 40min for brew deps install and musl-cross

• Explicitly install system libcurl on linux and osx

• This .cargo/config makes it test right locally: error: linking with cc failed: exit code: 1 rust-lang/rust#60149 (comment), only 5 tests failing now and no linking errors from libcurl (on OSX, at least) :)

• Cargo.toml missing quote misshap

• Same linker flags should apply for Linux builds? build-essential not necessary

• tid is i64 now, CRAM vs BAM test l_data is off by 3 and m_data is off too

• There should be no need for travis_wait with -v

• Debugging htslib structs with @jkbonfield guidance...

• Disable osx and musl-cross homebrew on the TravisCI matrix since it times out anyway, start looking into l_extranul and alignment issues for CRAM/BAM. Add curl_sys crate in hts-sys/src/lib.rs.

• Mystery solved for test_read_cram, testing some of the struct fields, explicitly state that l_data should differ according to Update to htslib 1.10.2 #184 (comment)

• Extract selective CRAM/BAM field comparison to test helper function

• Formatting

• Temporarily commenting the hfile format category detection makes .bed filetype tests pass but .BAI idx locator fails

• Format detection seems to be broken upstream on htslib, see Update to htslib 1.10.2 #184 (comment)

• Revert to libcurl4-openssl-dev instead of gnutls. All TravisCI script targets compiling and passing tests otherwise

• Bump up curl-sys version

• Experimenting musl cross-compilation with rustembedded cross. Ideally this would get rid of the homebrew musl-cross issue (too long compile times). /cc @nlhepler

• Remove assert_ne since cannot guarantee that l_data is different on all bam/crams

• Add minimum deps needed for rustembedded cross docker container

• Run docker in TravisCI, even if it is only supported on linux jobs :/

• Add CFLAGS suggested by @nlhepler

• Separate musl from GNU toolchains on Rust cross

• Bump up bindgen, vendorize openssl, further experimentation with compile flags

• Deprecate the mixed GNU/MUSL container

• Add minimal explanation on how to use the new docker files [ci skip]

• cross 0.2.0 with openssl vendored. Progress but more htslib compile time issues remain still

• Make sure libzma is present in both containers [ci skip]

• Focusing on LLVMv8 GNU cross build first, musl will come later on (if possible w/ htslib)

• Two more usize that should be u64

• Amend and settle BED header debate

• Rename test name accordingly

• Transition to cross instead of cargo for builds and tests

• All GNU/clang/llvm8 builds and tests seem to succeed, transitioning to musl testing now...

• Remove stray (repeated) musl build

• Remove @FiloSottile musl-cross homebrew formula since it takes too long to compile and timeouts CI, using clux/muslrust gets me a bit farther. Remove local rustup musl target... since htslib 1.10, the introduction of openssl complicated local compilation w/ musl greatly, unfortunately we have to resort to docker these days instead :/

• Experiment with toolchain setup in https://gitlab.com/rust_musl_docker/image. Remove vendored openssl since it's shipped in cross containers.

• Remove build.rs flags noise and different tests, containers should handle them transparently depending on the defined environment within, thanks @nlhepler

• Indicate dependencies and build flags required for OSX [ci skip].

• enable musl builds

• add libclang

• minor

• dbg

• revert

• dbg

• dbg

• Revert cross/docker madness, bottled up musl-cross myself over here: https://github.com/brainstorm/homebrew-musl-cross/commit/9304bf9cd6bc2b784083948771977d502030815a, high sierra bottle coming up soon. Cargo build script draft

• Add in openssl to hts-sys

• Make clippy boolean logic happy

• Remove homebrew OSX check since we'll just use rustembedded cross

Co-authored-by: Julian Gehring [email protected]
Co-authored-by: Johannes Köster [email protected]

[github-actions]

Pull Request Test Coverage Report for Build 9c327291c3e20e6590f3ab8f0fcd42d67654da21-PR-192

• 93 of 97 (95.88%) changed or added relevant lines in 8 files are covered.
• 2 unchanged lines in 2 files lost coverage.
• Overall coverage increased (+0.03%) to 92.932%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/bam/buffer.rs	1	2	50.0%
src/bam/mod.rs	29	30	96.67%
src/bam/record.rs	12	14	85.71%

Files with Coverage Reduction	New Missed Lines	%
src/bam/ext.rs	1	99.82%
src/tbx/mod.rs	1	79.61%

Totals
Change from base Build 03644609d6e3503491e878c50c684557187dbea9:	0.03%
Covered Lines:	10243
Relevant Lines:	11022

---

💛 - Coveralls