Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ant build of devel branch fails #43

Open
iquasere opened this issue May 24, 2019 · 31 comments
Open

Ant build of devel branch fails #43

iquasere opened this issue May 24, 2019 · 31 comments
Assignees

Comments

@iquasere
Copy link

Greetings

I used VizBin in the past, and it has never disappointed. It is a very good tool to visualize a spatial dispersion of the contigs, and very easy to install and use. However, I recently had to reformat my server, and when trying to reinstall VizBin as instructed in #15 , the ant build runs out of memory.

-verify-automatic-build:

-pre-pre-compile:

-pre-compile:

-copy-persistence-xml:

-compile-depend:

-do-compile:
    [javac] Compiling 45 source files to /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/classes
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 7

    [javac] 1 warning
    [javac]
    [javac]
    [javac] The system is out of resources.
    [javac] Consult the following stack trace for details.
    [javac] java.lang.OutOfMemoryError: Java heap space
    [javac]     at java.base/java.util.TimeZone.clone(TimeZone.java:753)
    [javac]     at java.base/sun.util.calendar.ZoneInfo.clone(ZoneInfo.java:639)
    [javac]     at java.base/java.util.TimeZone.getDefault(TimeZone.java:642)
    [javac]     at java.base/java.time.ZoneId.systemDefault(ZoneId.java:272)
    [javac]     at jdk.zipfs/jdk.nio.zipfs.ZipUtils.dosToJavaTime(ZipUtils.java:122)
    [javac]     at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$Entry.cen(ZipFileSystem.java:1960)
    [javac]     at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$Entry.readCEN(ZipFileSystem.java:1947)
    [javac]     at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getEntry(ZipFileSystem.java:1334)
    [javac]     at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:314)
    [javac]     at jdk.zipfs/jdk.nio.zipfs.ZipPath.getAttributes(ZipPath.java:727)
    [javac]     at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:293)
    [javac]     at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
    [javac]     at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
    [javac]     at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    [javac]     at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
    [javac]     at java.base/java.nio.file.Files.walkFileTree(Files.java:2760)
    [javac]     at jdk.compiler/com.sun.tools.javac.file.JavacFileManager$ArchiveContainer.<init>(JavacFileManager.java:520)
    [javac]     at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.getContainer(JavacFileManager.java:316)
    [javac]     at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:712)
    [javac]     at jdk.compiler/com.sun.tools.javac.code.ClassFinder.list(ClassFinder.java:734)
    [javac]     at jdk.compiler/com.sun.tools.javac.code.ClassFinder.scanUserPaths(ClassFinder.java:678)
    [javac]     at jdk.compiler/com.sun.tools.javac.code.ClassFinder.fillIn(ClassFinder.java:548)
    [javac]     at jdk.compiler/com.sun.tools.javac.code.ClassFinder.complete(ClassFinder.java:299)
    [javac]     at jdk.compiler/com.sun.tools.javac.code.ClassFinder$$Lambda$170/0x000000080018a040.complete(Unknown Source)
    [javac]     at jdk.compiler/com.sun.tools.javac.code.Symbol.complete(Symbol.java:642)
    [javac]     at jdk.compiler/com.sun.tools.javac.code.Symbol$PackageSymbol.members(Symbol.java:1131)
    [javac]     at jdk.compiler/com.sun.tools.javac.code.Symtab.listPackageModules(Symtab.java:832)
    [javac]     at jdk.compiler/com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:345)
    [javac]     at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:529)
    [javac]     at jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:286)
    [javac]     at jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:301)
    [javac]     at jdk.compiler/com.sun.tools.javac.comp.Enter.complete(Enter.java:576)

BUILD FAILED
/path/to/VizBin/src/interface/VizBin/nbproject/build-impl.xml:920: The following error occurred while executing this line:
/path/to/VizBin/src/interface/VizBin/nbproject/build-impl.xml:260: Compile failed; see the compiler error output for details.

Total time: 583 minutes 13 seconds

Since there have been no commits ever since I got this to work, I am wondering what could cause this problem. Has this been encountered when testing? I really would like to have this solved before integrating VizBin in my omics pipeline.

Thank you for your attention!

@claczny
Copy link
Owner

claczny commented May 24, 2019

Thank you for the kind words and we are very happy that VizBin has proven useful for your research!

@piotr-gawron could you please have a look a this? I am surprised by the `Total time: 583 minutes 13 seconds". IIRC, the compilations were typically quick and did not take almost 10 hours...

@piotr-gawron
Copy link
Collaborator

piotr-gawron commented May 24, 2019

@iquasere, @claczny
I think the problem is that you have openjdk-11 installed with "old" version of ant.
I was able to reproduce the problem on my ubuntu 16.04 machine using default packages:

  • java 11.0.1
  • ant 1.9.6

Problem was gone when I switched to:

  • java 1.8.0_212
  • ant 1.9.6

Of course I can imagine that you would like to use the latest java version. In such case you need to upgrade your ant. I tested it on ant 1.10.5 together with java 11.0.3 and it worked without problem (this is default for Ubuntu 18.04.2).

I wasn't able to pinpoint exactly in which version they fixed the problem, but between ant 1.9.6 and ant 1.10.5 there are some bug fixes related to java 11.

@iquasere
Copy link
Author

iquasere commented May 24, 2019

Damn, those were fast answers and troubleshooting!

The difference in versions would explain why it succeeded before, as the reformat was needed because of moving from 16.04 to 18.04. Problem is, ant is at the latest version, and still can't build VizBin. Java, however, is at 11.0.1, I'll update it and then attempt to build again

@piotr-gawron
Copy link
Collaborator

What do you mean ant is at the latest version? 1.10.5/1.9.14??

@iquasere
Copy link
Author

iquasere commented May 29, 2019

I am using the latest versions of the tools.

java version "11.0.2" 2019-01-15 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.2+9-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.2+9-LTS, mixed mode)
Apache Ant(TM) version 1.10.5 compiled on March 28 2019

Still, it runs out of memory. I know barely nothing about ant (although this troubles have pushed me to learn some things) and I get this warning:

[javac] warning: [options] bootstrap class path not set in conjunction with -source 7

@iquasere
Copy link
Author

I have also confirmed it uses version 11.0.2 of java.

print-version:
     [echo] Java/JVM version: 11
     [echo] Java/JVM detail version: 11.0.2

@piotr-gawron
Copy link
Collaborator

@iquasere,
One more question. It shouldn't be important, but what is your OS?

@iquasere
Copy link
Author

Ubuntu 18.04.2

@piotr-gawron
Copy link
Collaborator

dear @iquasere,
I cannot reproduce the error with this setup. I'm able to compile it on the clean ubuntu 18 system. You can try it by yourself using Ubuntu docker image (I just install necessary dependencies, clone repo and build VizBin interface):

docker run -t ubuntu sh -c "apt-get update; \
  apt-get install -y git ant openjdk-11-jdk-headless; \
  git clone https://github.com/claczny/VizBin.git; \
  cd VizBin/src/interface/VizBin/; \
  ant;"

However, I noticed one thing: the code that is cloned from repository contains 39 source files that are being compiled. But in the logs provided by you I see 45 source files. Maybe you introduced some code that makes maven cry?

Can you try the compilation with fresh clone from master branch?

@iquasere
Copy link
Author

iquasere commented Jun 5, 2019

I am cloning from devel branch, so I can use the CLI of VizBin. Does the master branch build allow for CLI usage?

@piotr-gawron
Copy link
Collaborator

Here is the command that shows that it's also working on devel branch:

docker run -t ubuntu sh -c "apt-get update; \
  apt-get install -y git ant openjdk-11-jdk-headless; \
  git clone https://github.com/claczny/VizBin.git; \
  cd VizBin; \
  git checkout devel; \
  cd src/interface/VizBin/; \
  ant;"

Have you tried to run the compilation on clean system? Maybe you installed something on your ubuntu that interferes with java/ant?

@iquasere
Copy link
Author

iquasere commented Jun 7, 2019

Well, I have some ideas on what the problems were, but it seems to be just bad configuration on my end. Running apt-get install -y openjdk-11-jdk-headless failed, and it reported to lock files in /var. After some random attempts at solving that, sudo ant finally did the trick. This probably was a mixture of setting different javas, because the problems on /var persist. Now it compiles in 10 seconds. As it should be.

Thank you very much for the valuable help! Gonna add it to my pipeline right now! Next publication, you gain one more reference ^^

@iquasere iquasere closed this as completed Jun 7, 2019
@iquasere
Copy link
Author

iquasere commented Jun 7, 2019

Well, the build went apparently fine.

Buildfile: /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build.xml

-pre-init:

-init-private:

-init-user:

-init-project:

-init-macrodef-property:

-do-init:

-post-init:

-init-check:

-init-ap-cmdline-properties:

-init-macrodef-javac-with-processors:

-init-macrodef-javac-without-processors:

-init-macrodef-javac:

-init-macrodef-test-impl:

-init-macrodef-junit-init:

-init-macrodef-junit-single:

-init-test-properties:

-init-macrodef-junit-batch:

-init-macrodef-junit:

-init-macrodef-junit-impl:

-init-macrodef-testng:

-init-macrodef-testng-impl:

-init-macrodef-test:

-init-macrodef-junit-debug:

-init-macrodef-junit-debug-batch:

-init-macrodef-junit-debug-impl:

-init-macrodef-test-debug-junit:

-init-macrodef-testng-debug:

-init-macrodef-testng-debug-impl:

-init-macrodef-test-debug-testng:

-init-macrodef-test-debug:

-init-debug-args:

-init-macrodef-nbjpda:

-init-macrodef-debug:

-init-macrodef-java:

-init-presetdef-jar:

-init-ap-cmdline-supported:

-init-ap-cmdline:

init:

-deps-jar-init:

deps-jar:
    [mkdir] Created dir: /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build

-warn-already-built-jar:
[propertyfile] Updating property file: /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/built-jar.properties

-check-automatic-build:

-clean-after-automatic-build:

-verify-automatic-build:

-pre-pre-compile:
    [mkdir] Created dir: /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/classes

-pre-compile:

-copy-persistence-xml:

-compile-depend:

-do-compile:
    [mkdir] Created dir: /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/empty
    [mkdir] Created dir: /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/generated-sources/ap-source-output
    [javac] Compiling 45 source files to /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/classes
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 7
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] 1 warning
    [javac] Creating empty /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/classes/lu/uni/lcsb/vizbin/clustering/package-info.class
    [javac] Creating empty /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/classes/lu/uni/lcsb/vizbin/graphics/package-info.class
    [javac] Creating empty /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/classes/lu/uni/lcsb/vizbin/data/package-info.class
    [javac] Creating empty /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/classes/lu/uni/lcsb/vizbin/pca/package-info.class
    [javac] Creating empty /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/classes/lu/uni/lcsb/vizbin/package-info.class
    [javac] Creating empty /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/classes/lu/uni/lcsb/vizbin/utils/package-info.class
     [copy] Copying 1 file to /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build/classes

-post-compile:

compile:

-pre-jar:

-pre-pre-jar:

-do-jar-create-manifest:

-do-jar-copy-manifest:
     [copy] Copying 1 file to /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/build

-do-jar-set-mainclass:

-do-jar-set-profile:

-do-jar-set-splashscreen:

-do-jar-jar:
[j2seproject1:jar] Building jar: /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/dist/VizBin.jar
     [echo] To run this application from the command line without Ant, try:
     [echo] /usr/lib/jvm/java-11-openjdk-amd64/bin/java -cp /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/blas.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/commons-io-2.4.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/commons-cli-1.2.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/ejml-0.23.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/mtj-1.0.2-SNAPSHOT-jar-with-dependencies.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/hamcrest-core-1.3.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/jcommon-1.0.23.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/jfreechart-1.0.19-experimental.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/jfreechart-1.0.19-swt.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/jfreechart-1.0.19.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/jfreesvg-2.0.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/orsoncharts-1.4-eval-nofx.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/orsonpdf-1.6-eval.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/servlet.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/swtgraphics2d.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/lib/log4j-1.2.17.jar:/mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/dist/VizBin.jar lu.uni.lcsb.vizbin.Main

-init-macrodef-copylibs:

-do-jar-copylibs:

-do-jar-delete-manifest:

-do-jar-without-libraries:

-do-jar-with-libraries:

-post-jar:
      [jar] Building jar: /mnt/HDDStorage/jsequeira/VizBin/src/interface/VizBin/dist/VizBin-dist.jar

-do-jar:

jar:

BUILD SUCCESSFUL
Total time: 8 seconds

But when running VizBin...

2019-06-07 10:56:11,857 DEBUG [main] (Main.java:43) - Running command line...
2019-06-07 10:56:11,865 DEBUG [main] (ProcessInput.java:63) - Init of ProcessInput
2019-06-07 10:56:11,871 DEBUG [Thread-0] (ObjectWithProperties.java:33) - Property changed: FINISHED. Old: true New: false
2019-06-07 10:56:11,871 DEBUG [Thread-0] (ProcessInput.java:108) - /tmp
2019-06-07 10:56:11,895 DEBUG [Thread-0] (ProcessInput.java:117) - Loading data from file.
Contig length treshold: 1000
2019-06-07 10:56:11,895 DEBUG [Thread-0] (ProcessInput.java:78) - Loading fasta file: debugMOSCA/Assembly/4478-DNA-S1613-MiSeqKapa/contigs.fasta
2019-06-07 10:56:11,917 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Loading fasta file: debugMOSCA/Assembly/4478-DNA-S1613-MiSeqKapa/contigs.fasta (5)
2019-06-07 10:56:11,996  INFO [Thread-0] (DataSetFactory.java:302) - Number of contigs longer or equal then 1000: 199
Number of contigs shorter then 1000: 6283
2019-06-07 10:56:12,001 DEBUG [Thread-0] (ProcessInput.java:78) - DataSet loaded (199 sequences)
2019-06-07 10:56:12,002 DEBUG [Thread-0] (ProcessInput.java:78) - Creating kmers (k=5, merge = true)
2019-06-07 10:56:12,002 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] DataSet loaded (199 sequences) (5)
2019-06-07 10:56:12,002 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Creating kmers (k=5, merge = true) (5)
2019-06-07 10:56:12,004 DEBUG [Thread-0] (DataSetUtils.java:76) - Creating usedVal for k=5
2019-06-07 10:56:12,055 DEBUG [Thread-0] (ProcessInput.java:78) - Normalizing vectors...
2019-06-07 10:56:12,055 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Normalizing vectors... (10)
2019-06-07 10:56:12,063 DEBUG [Thread-0] (ProcessInput.java:78) - Clr normalization...
2019-06-07 10:56:12,065 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Clr normalization... (15)
2019-06-07 10:56:12,080 DEBUG [Thread-0] (ProcessInput.java:78) - Running PCA... (Mtj)
2019-06-07 10:56:12,080 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Running PCA... (Mtj) (20)
Jun 07, 2019 10:56:12 AM com.github.fommil.netlib.LAPACK <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
Jun 07, 2019 10:56:12 AM com.github.fommil.netlib.LAPACK <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
2019-06-07 10:56:13,159 DEBUG [Thread-0] (DataSetUtils.java:256) - DONE: Computed the new basis.
2019-06-07 10:56:13,230 DEBUG [Thread-0] (DataSetUtils.java:263) - DONE: Projected from sample to eigen space.
2019-06-07 10:56:13,230 DEBUG [Thread-0] (ProcessInput.java:78) - Running T-SNE...
2019-06-07 10:56:13,231 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Running T-SNE... (35)
2019-06-07 10:56:13,259 DEBUG [Thread-0] (DataSetUtils.java:271) - Running command: "/home/jsequeira/.vizbin/pbh_tsne" in directory: /tmp/map497744776097839557
Number of threads: 12
Seed: 0
java.io.IOException: Cannot run program "/home/jsequeira/.vizbin/pbh_tsne" (in directory "/tmp/map497744776097839557"): error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at java.lang.Runtime.exec(Runtime.java:620)
        at lu.uni.lcsb.vizbin.service.utils.DataSetUtils$TSNERunner.run(DataSetUtils.java:293)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 3 more
java.io.FileNotFoundException: points.txt file not found. Probably bh_tsne binaries execution failed.
        at lu.uni.lcsb.vizbin.ProcessInput$2.run(ProcessInput.java:155)
2019-06-07 10:56:13,337 DEBUG [Thread-0] (ProcessInput.java:78) - Error! Check the logs.
2019-06-07 10:56:13,337 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Error! Check the logs. (35)
2019-06-07 10:56:13,337 ERROR [Thread-0] (ProcessInput.java:224) - points.txt file not found. Probably bh_tsne binaries execution failed.
java.io.FileNotFoundException: points.txt file not found. Probably bh_tsne binaries execution failed.
        at lu.uni.lcsb.vizbin.ProcessInput$2.run(ProcessInput.java:155)
2019-06-07 10:56:13,340 DEBUG [Thread-0] (ObjectWithProperties.java:33) - Property changed: FINISHED. Old: false New: true

This means the ant build still didn't finished properly, right? A reformat on the system seems the only solution at this point...

@iquasere iquasere reopened this Jun 7, 2019
@iquasere
Copy link
Author

iquasere commented Jun 7, 2019

Attempting to use the docker run also didn't work

BUILD SUCCESSFUL
Total time: 11 seconds
2019-06-07 13:08:15,668 DEBUG [main] (Main.java:43) - Running command line...
java.io.FileNotFoundException: /root/.vizbin/config (No such file or directory)
        at java.base/java.io.FileInputStream.open0(Native Method)
        at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
        at lu.uni.lcsb.vizbin.Settings.loadSettings(Settings.java:111)
        at lu.uni.lcsb.vizbin.Main.main(Main.java:44)
2019-06-07 13:08:15,686 DEBUG [main] (ProcessInput.java:63) - Init of ProcessInput
2019-06-07 13:08:15,692 DEBUG [Thread-0] (ObjectWithProperties.java:33) - Property changed: FINISHED. Old: true New: false
2019-06-07 13:08:15,693 DEBUG [Thread-0] (ProcessInput.java:108) - /tmp
2019-06-07 13:08:15,718 DEBUG [Thread-0] (ProcessInput.java:117) - Loading data from file.
Contig length treshold: 1000
2019-06-12 16:43:59,441 DEBUG [Thread-0] (ProcessInput.java:78) - Loading fasta file: /input/4478-DNA-S1613-MiSeqKapa/contigs.fasta
2019-06-12 16:43:59,516 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Loading fasta file: /input/4478-DNA-S1613-MiSeqKapa/contigs.fasta (5)
2019-06-12 16:43:59,614  INFO [Thread-0] (DataSetFactory.java:302) - Number of contigs longer or equal then 1000: 199
Number of contigs shorter then 1000: 6285
2019-06-12 16:43:59,622 DEBUG [Thread-0] (ProcessInput.java:78) - DataSet loaded (199 sequences)
2019-06-12 16:43:59,623 DEBUG [Thread-0] (ProcessInput.java:78) - Creating kmers (k=5, merge = true)
2019-06-12 16:43:59,623 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] DataSet loaded (199 sequences) (5)
2019-06-12 16:43:59,623 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Creating kmers (k=5, merge = true) (5)
2019-06-12 16:43:59,624 DEBUG [Thread-0] (DataSetUtils.java:76) - Creating usedVal for k=5
2019-06-12 16:43:59,689 DEBUG [Thread-0] (ProcessInput.java:78) - Normalizing vectors...
2019-06-12 16:43:59,690 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Normalizing vectors... (10)
2019-06-12 16:43:59,698 DEBUG [Thread-0] (ProcessInput.java:78) - Clr normalization...
2019-06-12 16:43:59,700 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Clr normalization... (15)
2019-06-12 16:43:59,713 DEBUG [Thread-0] (ProcessInput.java:78) - Running PCA... (Mtj)
2019-06-12 16:43:59,713 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Running PCA... (Mtj) (20)
Jun 12, 2019 4:43:59 PM com.github.fommil.netlib.LAPACK <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
Jun 12, 2019 4:43:59 PM com.github.fommil.netlib.LAPACK <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
2019-06-12 16:44:00,897 DEBUG [Thread-0] (DataSetUtils.java:256) - DONE: Computed the new basis.
2019-06-12 16:44:00,990 DEBUG [Thread-0] (DataSetUtils.java:263) - DONE: Projected from sample to eigen space.
2019-06-12 16:44:00,991 DEBUG [Thread-0] (ProcessInput.java:78) - Running T-SNE...
2019-06-12 16:44:00,991 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Running T-SNE... (35)
2019-06-12 16:44:01,027 DEBUG [Thread-0] (DataSetUtils.java:271) - Running command: "null" in directory: /tmp/map14198620525788122565
Number of threads: 1
Seed: 0
java.lang.NullPointerException
        at lu.uni.lcsb.vizbin.service.utils.DataSetUtils$TSNERunner.run(DataSetUtils.java:293)
        at java.base/java.lang.Thread.run(Thread.java:834)
java.io.FileNotFoundException: points.txt file not found. Probably bh_tsne binaries execution failed.
        at lu.uni.lcsb.vizbin.ProcessInput$2.run(ProcessInput.java:155)
2019-06-12 16:44:01,029 DEBUG [Thread-0] (ProcessInput.java:78) - Error! Check the logs.
2019-06-12 16:44:01,029 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Error! Check the logs. (35)
2019-06-12 16:44:01,030 ERROR [Thread-0] (ProcessInput.java:224) - points.txt file not found. Probably bh_tsne binaries execution failed.
java.io.FileNotFoundException: points.txt file not found. Probably bh_tsne binaries execution failed.
        at lu.uni.lcsb.vizbin.ProcessInput$2.run(ProcessInput.java:155)
2019-06-12 16:44:01,032 DEBUG [Thread-0] (ObjectWithProperties.java:33) - Property changed: FINISHED. Old: false New: true

@piotr-gawron
Copy link
Collaborator

Well, that's completely different question :). You explicitly asked about problems with ant build ;).

Anyway, the full build process should start from top directory using make command. If you run it directly from the cloned repository it will probably not work due to:

I think Cedric forgot to document it...

@iquasere
Copy link
Author

Well, it started as an installation problem, but it has evolved xD Now, some months ago I ran the commands as they were specified in the issue concerning use through the command line. At that time, the tool was easily installed and used. Could it be that running it in Ubuntu 16.04 made the difference? Or maybe my system was already equiped with the libraries that it doesn't install by running ant from a sub-directory.

@piotr-gawron
Copy link
Collaborator

What made a difference is that before at the beginning you probably run the distribution jar (https://github.com/claczny/VizBin/blob/master/VizBin-dist.jar) which contained tsne. When run for the first time it extracts tsne into local folder on your computer. And after that you run the code from 'devel' branch. This compilation didn't contain tsne, however the tsne was already on your computer.

Now, I assume that you are trying to compile and run it on clean machine. Therefore tsne is neither in the jar nor in the local system. And you get an error.

@iquasere
Copy link
Author

I don't think that is what is happening.

find ./VizBin/ -name "*tsne*"

finds some files

./VizBin/src/interface/web-interface/bin/lcsb/binning/tsne
./VizBin/src/interface/web-interface/build/web/WEB-INF/classes/lcsb/binning/tsne
./VizBin/src/interface/web-interface/src/lcsb/binning/tsne
./VizBin/src/interface/web-interface/test/lcsb/binning/tsne
./VizBin/src/backend/bh_tsne
./VizBin/src/backend/bh_tsne/tsne.h
./VizBin/src/backend/bh_tsne/tsne.cpp
./VizBin/src/backend/bh_tsne/ptsne.cpp

Aren't these the files required?

@piotr-gawron
Copy link
Collaborator

And none of them is a compiled file for your OS. This is how it looks like after compilation for windows and linux:

gawi@uni-new:~/workspace$ find ./VizBin/ -name "*tsne*"
./VizBin/src/interface/VizBin/src/tsne
./VizBin/src/interface/VizBin/src/tsne/pbh_tsne
./VizBin/src/interface/VizBin/src/tsne/pbh_tsne.exe
./VizBin/src/interface/VizBin/build/classes/tsne
./VizBin/src/interface/VizBin/build/classes/tsne/pbh_tsne
./VizBin/src/interface/VizBin/build/classes/tsne/pbh_tsne.exe
./VizBin/src/interface/web-interface/src/lcsb/binning/tsne
./VizBin/src/interface/web-interface/build/web/WEB-INF/classes/lcsb/binning/tsne
./VizBin/src/interface/web-interface/bin/lcsb/binning/tsne
./VizBin/src/interface/web-interface/test/lcsb/binning/tsne
./VizBin/src/backend/bh_tsne
./VizBin/src/backend/bh_tsne/pbh_tsne
./VizBin/src/backend/bh_tsne/tsne.h
./VizBin/src/backend/bh_tsne/tsne.cpp
./VizBin/src/backend/bh_tsne/pbh_tsne.exe
./VizBin/src/backend/bh_tsne/ptsne.cpp

@claczny
Copy link
Owner

claczny commented Jun 21, 2019

@iquasere Not sure if the issue related to @piotr-gawron s comment

I think Cedric forgot to document it...

persists, but have you looked into https://github.com/claczny/VizBin/wiki/Minimal-Build-Environment already?

Also, if there is anything you think could be made clearer in the documentation, please let us know - preferably in a separate issue ;)

@iquasere
Copy link
Author

iquasere commented Jun 21, 2019

Can you please confirm me if this works in your system?

docker run -v /folder_with_contigs/:/input -v /folder_for_output/:/output -t ubuntu sh -c "apt-get update; \
apt-get install -y git ant openjdk-11-jdk-headless; \
git clone https://github.com/claczny/VizBin.git; \
cd VizBin; \
git checkout devel; \
cd src/interface/VizBin/; \
ant; \
cd ../../..; \
java -jar VizBin-dist.jar -i /input/contigs.fasta -o /output/points.txt"

Must be used with some contigs.fasta file, but you probably have some available.

@piotr-gawron
Copy link
Collaborator

It doesn't work and as I mentioned before it shouldn't work. You don't compile the whole project with this command, but only java part. If you want to make it working like that try this, but it's hacky:

docker run -v /folder_with_contigs/:/input   -v /folder_for_output/:/output  -t ubuntu sh -c "apt-get update; \
apt-get install -y git ant openjdk-11-jdk-headless; \
git clone https://github.com/claczny/VizBin.git; \
cd VizBin; \

jar xf VizBin-dist.jar; \
mv tsne src/interface/VizBin/src/; \
rm VizBin-dist.jar ;/

git checkout devel; \
cd src/interface/VizBin/; \
ant; \
cd ../../..; \
java -jar src/interface/VizBin/dist/VizBin-dist.jar -i data/EqualSet01.fa -o points.txt"

There are three differences here:

  1. I unpack original VizBin-dist.jar file with compiled tsne and put it in the proper folder
  2. I call the VizBin-dist.jar created by ant (the one you are calling is created by Makefile you don't call)
  3. I call it on the input file provided in the repo, because I don't have access to your data

@iquasere
Copy link
Author

iquasere commented Jun 25, 2019

This

mv tsne src/interface/VizBin/src/;

was the part I was understanding, but had no knowledge to know how to do. But still, it does not work! Exactly same error as before! What do you mean by compiling only the java part? What is the docker command that makes VizBin work through the command line?

@claczny
Copy link
Owner

claczny commented Jun 25, 2019

Let me try to clarify this :)

VizBin combines a frontend (the data input/preprocessing and visualization component) with a backend (a C/C++-based program that implements BH-SNE).

The frontend is programmed in Java and needs to be compiled using ant.

The backend is programmed in C/C++ and needs to be compiled with a C/C++ compiler. These compilers differ by operating system, e.g, g++ from the Gnu Compiler Collection (GCC) in Linux. Compiling the C/C++ code results in a binary, which needs to be created for every OS separately. This is handled by the Makefile.

Without these binaries, VizBin is incomplete. Hence, it is not sufficient to run the Java-compilation, but the C/C++-compilation should/must also occur. To facilitate this process, we have provided the Minimal Build Environment instructions. Have you looked into these?

Moreover, I have to admit that I am a bit confused about the docker part that you are posting. Where did you find the respective descriptions?

Hope this helps.

Best,

Cedric

@iquasere
Copy link
Author

The error persists if I do

git clone https://github.com/claczny/VizBin.git
bash VizBin/setupUbuntu.sh

sudo apt-get update
sudo apt-get install -y git ant openjdk-11-jdk-headless
cd VizBin

jar xf VizBin-dist.jar
mv tsne src/interface/VizBin/src/
rm VizBin-dist.jar

git checkout devel
cd src/interface/VizBin/
ant
cd ../../..
java -jar VizBin-dist.jar -i /path/to/contigs.fasta -o points.txt

@claczny
Copy link
Owner

claczny commented Jun 25, 2019

Maybe a strange question, but have you tried simply running make in the top-directory of the repository of the devel branch?

Your build process seems to be overly explicit.
Put differently, the Makefiles should handle the build unless one has a diverging setup, which should not be the case when using the Minimal Builld Environment.

@iquasere
Copy link
Author

iquasere commented Jun 25, 2019

Run make after all the previous commands, or on a fresh cloned repo? The error persists if simply

cd src/interface/VizBin/
make
cd ../../..
java -jar VizBin-dist.jar -i contigs.fasta -o points.txt

@claczny
Copy link
Owner

claczny commented Jun 25, 2019

I would suggest a clean checkout -> switch to devel branch -> ensure you have the Minimal Build Environment set up -> make in the top directory.

@iquasere
Copy link
Author

Does not work as

git clone https://github.com/claczny/VizBin.git
bash VizBin/setupUbuntu.sh
git checkout devel
cd src/interface/VizBin/
make
cd ../../..
java -jar VizBin-dist.jar -i /path/to/contigs.fasta -o points.txt

@claczny
Copy link
Owner

claczny commented Jun 25, 2019

Could you please share if there are any errors/warnings reported and, if so, which?
Because I would expect the whole process to produce some error/warning then at some point.

@iquasere
Copy link
Author

The error reported has been the same every time

2019-06-25 10:52:28,136 DEBUG [main] (Main.java:43) - Running command line...
2019-06-25 10:52:28,143 DEBUG [main] (ProcessInput.java:63) - Init of ProcessInput
2019-06-25 10:52:28,149 DEBUG [Thread-0] (ObjectWithProperties.java:33) - Property changed: FINISHED. Old: true New: false
2019-06-25 10:52:28,150 DEBUG [Thread-0] (ProcessInput.java:108) - /tmp
2019-06-25 10:52:28,172 DEBUG [Thread-0] (ProcessInput.java:117) - Loading data from file.
Contig length treshold: 1000
2019-06-25 10:52:28,172 DEBUG [Thread-0] (ProcessInput.java:78) - Loading fasta file: /mnt/HDDStorage/jsequeira/debugMOSCA/Assembly/4478-DNA-S1613-MiSeqKapa/contigs.fasta
2019-06-25 10:52:28,194 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Loading fasta file: /mnt/HDDStorage/jsequeira/debugMOSCA/Assembly/4478-DNA-S1613-MiSeqKapa/contigs.fasta (5)
2019-06-25 10:52:28,418  INFO [Thread-0] (DataSetFactory.java:302) - Number of contigs longer or equal then 1000: 259
Number of contigs shorter then 1000: 41114
2019-06-25 10:52:28,424 DEBUG [Thread-0] (ProcessInput.java:78) - DataSet loaded (259 sequences)
2019-06-25 10:52:28,425 DEBUG [Thread-0] (ProcessInput.java:78) - Creating kmers (k=5, merge = true)
2019-06-25 10:52:28,425 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] DataSet loaded (259 sequences) (5)
2019-06-25 10:52:28,425 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Creating kmers (k=5, merge = true) (5)
2019-06-25 10:52:28,427 DEBUG [Thread-0] (DataSetUtils.java:76) - Creating usedVal for k=5
2019-06-25 10:52:28,490 DEBUG [Thread-0] (ProcessInput.java:78) - Normalizing vectors...
2019-06-25 10:52:28,490 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Normalizing vectors... (10)
2019-06-25 10:52:28,499 DEBUG [Thread-0] (ProcessInput.java:78) - Clr normalization...
2019-06-25 10:52:28,502 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Clr normalization... (15)
2019-06-25 10:52:28,520 DEBUG [Thread-0] (ProcessInput.java:78) - Running PCA... (Mtj)
2019-06-25 10:52:28,520 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Running PCA... (Mtj) (20)
Jun 25, 2019 10:52:28 AM com.github.fommil.netlib.LAPACK <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
Jun 25, 2019 10:52:28 AM com.github.fommil.netlib.LAPACK <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
2019-06-25 10:52:29,629 DEBUG [Thread-0] (DataSetUtils.java:256) - DONE: Computed the new basis.
2019-06-25 10:52:29,708 DEBUG [Thread-0] (DataSetUtils.java:263) - DONE: Projected from sample to eigen space.
2019-06-25 10:52:29,708 DEBUG [Thread-0] (ProcessInput.java:78) - Running T-SNE...
2019-06-25 10:52:29,709 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Running T-SNE... (35)
2019-06-25 10:52:29,743 DEBUG [Thread-0] (DataSetUtils.java:271) - Running command: "/home/jsequeira/.vizbin/pbh_tsne" in directory: /tmp/map1902435464157112917
Number of threads: 1
Seed: 0
java.io.IOException: Cannot run program "/home/jsequeira/.vizbin/pbh_tsne" (in directory "/tmp/map1902435464157112917"): error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at java.lang.Runtime.exec(Runtime.java:620)
        at lu.uni.lcsb.vizbin.service.utils.DataSetUtils$TSNERunner.run(DataSetUtils.java:293)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 3 more
java.io.FileNotFoundException: points.txt file not found. Probably bh_tsne binaries execution failed.
        at lu.uni.lcsb.vizbin.ProcessInput$2.run(ProcessInput.java:155)
2019-06-25 10:52:29,821 DEBUG [Thread-0] (ProcessInput.java:78) - Error! Check the logs.
2019-06-25 10:52:29,822 DEBUG [AWT-EventQueue-0] (ProcessInput.java:94) - [PROGRESS BAR] Error! Check the logs. (35)
2019-06-25 10:52:29,822 ERROR [Thread-0] (ProcessInput.java:224) - points.txt file not found. Probably bh_tsne binaries execution failed.
java.io.FileNotFoundException: points.txt file not found. Probably bh_tsne binaries execution failed.
        at lu.uni.lcsb.vizbin.ProcessInput$2.run(ProcessInput.java:155)
2019-06-25 10:52:29,825 DEBUG [Thread-0] (ObjectWithProperties.java:33) - Property changed: FINISHED. Old: false New: true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants