Train and testing quickstart Indian model fails

[sidhantsetia]

I successfully built Joshua but I'm unable to train the quick start Indian model. The file build_binary is not present. I'm trying to build on ubuntu 14.04
The error message is added below.
After the build /joshua/src/joshua/decoder/ff/lm/kenlm this folder is not present.

$JOSHUA/bin/pipeline.pl --source bn --target en \

--type hiero \
--no-prepare --aligner berkeley \
--corpus input/bn-en/tok/training.bn-en \
--tune input/bn-en/tok/dev.bn-en \
--test input/bn-en/tok/devtest.bn-en

[source-numlines] rebuilding...
dep=/home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.bn [CHANGED]
cmd=cat /home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.bn | wc -l
took 0 seconds (0s)
[source-numlines] retrieved cached result => 20788
[berkeley-aligner-chunk-0] rebuilding...
dep=alignments/0/word-align.conf [CHANGED]
dep=/home/ssetia/models/bn-en/data/train/splits/corpus.bn.0 [CHANGED]
dep=/home/ssetia/models/bn-en/data/train/splits/corpus.en.0 [CHANGED]
dep=alignments/0/training.align [NOT FOUND]
cmd=java -d64 -Xmx10g -jar /home/ssetia/dr/joshua/lib/berkeleyaligner.jar ++alignments/0/word-align.conf
took 199 seconds (3m19s)
[aligner-combine] rebuilding...
dep=alignments/0/training.align [CHANGED]
dep=alignments/training.align [NOT FOUND]
cmd=cat alignments/0/training.align > alignments/training.align
took 0 seconds (0s)
[thrax-input-file] rebuilding...
dep=/home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.bn [CHANGED]
dep=/home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en [CHANGED]
dep=alignments/training.align [CHANGED]
dep=/home/ssetia/models/bn-en/data/train/thrax-input-file [NOT FOUND]
cmd=/home/ssetia/dr/joshua/scripts/training/paste /home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.bn /home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en alignments/training.align | perl -pe 's/\t/ ||| /g' | grep -v '()' | grep -v '||| +$' > /home/ssetia/models/bn-en/data/train/thrax-input-file
took 0 seconds (0s)
[thrax-prep] rebuilding...
dep=/home/ssetia/models/bn-en/data/train/thrax-input-file [CHANGED]
dep=grammar.gz [NOT FOUND]
cmd=hadoop/bin/hadoop fs -rm -r pipeline-bn-en-hiero-_home_ssetia_models_bn-en; hadoop/bin/hadoop fs -mkdir pipeline-bn-en-hiero-_home_ssetia_models_bn-en; hadoop/bin/hadoop fs -put /home/ssetia/models/bn-en/data/train/thrax-input-file pipeline-bn-en-hiero-_home_ssetia_models_bn-en/input-file
took 2 seconds (2s)
[thrax-run] rebuilding...
dep=/home/ssetia/models/bn-en/data/train/thrax-input-file [CHANGED]
dep=thrax-hiero.conf [CHANGED]
dep=grammar.gz [NOT FOUND]
cmd=hadoop/bin/hadoop jar /home/ssetia/dr/joshua/thrax/bin/thrax.jar -D mapred.child.java.opts='-Xmx2g' -D hadoop.tmp.dir=/tmp thrax-hiero.conf pipeline-bn-en-hiero-_home_ssetia_models_bn-en > thrax.log 2>&1; rm -f grammar grammar.gz; hadoop/bin/hadoop fs -getmerge pipeline-bn-en-hiero-_home_ssetia_models_bn-en/final/ grammar.gz
took 98 seconds (1m38s)
[lm-sort-uniq] rebuilding...
dep=/home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en [CHANGED]
dep=/home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en.uniq [NOT FOUND]
cmd=/home/ssetia/dr/joshua/scripts/training/scat /home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en | sort -u -T /tmp -S 2G | gzip -9n > /home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en.uniq
took 0 seconds (0s)
[kenlm] rebuilding...
dep=/home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en.uniq [CHANGED]
dep=lm.gz [NOT FOUND]
cmd=/home/ssetia/dr/joshua/bin/lmplz -o 5 -T /tmp -S 2G --verbose_header --text /home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en.uniq | gzip -9n > lm.gz
took 0 seconds (0s)
[compile-kenlm] rebuilding...
dep=lm.gz [CHANGED]
dep=lm.kenlm [NOT FOUND]
cmd=/home/ssetia/dr/joshua/src/joshua/decoder/ff/lm/kenlm/build_binary lm.gz lm.kenlm
JOB FAILED (return code 127)
/bin/bash: /home/ssetia/dr/joshua/src/joshua/decoder/ff/lm/kenlm/build_binary: No such file or directory

[mjpost]

KenLM didn't build. Type "ant kenlm".

What version of Joshua are you using? Development or the 6.0.4 release?

[sidhantsetia]

I'm using the development branch as the 6.0.4 was giving build failure because of boost.
I ran ant kenlm and this is the error I am getting

ant kenlm
Buildfile: /home/ssetia/dr/joshua/build.xml

check-joshua-home:
[echo] JOSHUA = /home/ssetia/dr/joshua basedir = /home/ssetia/dr/joshua

kenlm:
[exec] -- Boost version: 1.59.0
[exec] -- Found the following Boost libraries:
[exec] -- program_options
[exec] -- system
[exec] -- thread
[exec] -- unit_test_framework
[exec] -- Configuring done
[exec] -- Generating done
[exec] -- Build files have been written to: /home/ssetia/dr/joshua/src/kenlm/build
[exec] [ 35%] Built target kenlm_util
[exec] [ 37%] Built target bit_packing_test
[exec] [ 38%] Built target file_piece_test
[exec] [ 39%] Built target joint_sort_test
[exec] [ 41%] Built target multi_intersection_test
[exec] [ 42%] Built target probing_hash_table_test
[exec] [ 43%] Built target read_compressed_test
[exec] [ 44%] Built target sorted_uniform_test
[exec] [ 46%] Built target tokenize_piece_test
[exec] [ 47%] Built target io_test
[exec] [ 48%] Built target sort_test
[exec] [ 50%] Built target stream_test
[exec] [ 69%] Built target kenlm
[exec] [ 70%] Built target build_binary
[exec] [ 71%] Built target fragment
[exec] [ 73%] Built target left_test
[exec] [ 74%] Built target model_test
[exec] [ 75%] Built target partial_test
[exec] [ 76%] Built target query
[exec] [ 82%] Built target kenlm_common
[exec] [ 89%] Built target kenlm_builder
[exec] [ 91%] Built target adjust_counts_test
[exec] [ 92%] Built target corpus_count_test
[exec] [ 93%] Built target lmplz
[exec] [ 97%] Built target kenlm_filter
[exec] [ 98%] Built target filter
[exec] [100%] Built target phrase_table_vocab
[exec] /usr/bin/ld: lm/CMakeFiles/kenlm.dir/bhiksha.cc.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
[exec] lm/CMakeFiles/kenlm.dir/bhiksha.cc.o: error adding symbols: Bad value
[exec] collect2: error: ld returned 1 exit status
[exec] Result: 1

I'm using the latest cmake and I'm trying to build on ubuntu 14.04

[mjpost]

Hmm. I'm not sure, since this compiles on both OS X and CentOS 6.7 for me.

One suggestion: edit $JOSHUA/jni/build_kenlm.sh, and try removing the "-fPIC" line. You could also try to add the CXXFLAGS to the cmake invocation.

Make sure to run "ant clean-kenlm" before "ant kenlm", and please report back here how things go!

[sidhantsetia]

I tried to build again by removing the "-fPIC" line. But I'm still getting an error.

[exec] read_compressed.cc:(.text+0xb7f): undefined reference to `inflate'
[exec] collect2: error: ld returned 1 exit status
[exec] make[2]: *** [bin/bit_packing_test] Error 1
[exec] make[1]: *** [util/CMakeFiles/bit_packing_test.dir/all] Error 2
[exec] make: *** [all] Error 2
[exec] cp: cannot stat ‘bin/query’: No such file or directory
[exec] cp: cannot stat ‘bin/lmplz’: No such file or directory
[exec] cp: cannot stat ‘bin/build_binary’: No such file or directory
[exec] Linking CXX executable ../bin/bit_packing_test
[exec] util/CMakeFiles/bit_packing_test.dir/build.make:145: recipe for target 'bin/bit_packing_test' failed
[exec] CMakeFiles/Makefile2:964: recipe for target 'util/CMakeFiles/bit_packing_test.dir/all' failed
[exec] Makefile:126: recipe for target 'all' failed
[exec] g++: error: lm/CMakeFiles/kenlm.dir/*.o: No such file or directory
[exec] Result: 1

[sidhantsetia]

Hi @mjpost

I was able to build kenlm by adding the CXXFLAGS to cmake invocation. I build joshua after that using ant java. It gave a build success.

However, when I try to train and test the Indian model it still gives the same error as above.

[mjpost]

Oh, the path is wrong. build_binary is in $JOSHUA/bin, not in $JOSHUA/src/joshua/decoder/ff/lm/kenlm. I just pushed up a fix, so if you're using the development version, "git pull" should fix it. Otherwise, you can fix the path manually in $JOSHUA/bin/pipeline.pl.

Can you tell me the exact changes you made to $JOSHUA/jni/build_kenlm.sh, so that I can see if they should be added to Joshua?

[sidhantsetia]

@mjpost

I did not make any changes to $JOSHUA/jni/build_kenlm.sh. I declared an environment variable for CXX flags.
I also made a change to , where I added $JOSHUA/src/kenlm/CMakeLists.txt
set(CMAKE_CXX_FLAGS "-fPIC")

[sidhantsetia]

After the new pull I'm getting build fail in giza.

Here are the logs that I am getting
giza:
[exec] make: Entering directory '/home/ssetia/dr/joshua/src/giza-pp'In file included from /usr/include/c++/4.9/unordered_map:35:0,
[exec] /home/ssetia/dr/joshua
[exec] from mystl.h:22,
[exec]
[exec] make -C GIZA++-v2
[exec] from myassert.cpp:1:/home/ssetia/dr/joshua
[exec]
[exec] make -C mkcls-v2
[exec] make[1]: Entering directory '/home/ssetia/dr/joshua/src/giza-pp/GIZA++-v2'
[exec] /usr/include/c++/4.9/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
[exec] #error This file requires compiler and library support for the
[exec] ^g++ -Wall -Wno-parentheses -O3 -funroll-loops -DNDEBUG -DWORDINDEX_WITH_4_BYTE -DBINARY_SEARCH_FOR_TTABLE -DWORDINDEX_WITH_4_BYTE -c Parameter.cpp -o optimized/Parameter.o
[exec]
[exec] In file included from /usr/include/c++/4.9/unordered_map:35:0,g++ -Wall -Wno-parentheses -O3 -funroll-loops -DNDEBUG -DWORDINDEX_WITH_4_BYTE -DBINARY_SEARCH_FOR_TTABLE -DWORDINDEX_WITH_4_BYTE -c myassert.cpp -o optimized/myassert.o
[exec]
[exec] from mystl.h:22,
[exec] from Parameter.h:26,g++ -Wall -Wno-parentheses -O3 -funroll-loops -DNDEBUG -DWORDINDEX_WITH_4_BYTE -DBINARY_SEARCH_FOR_TTABLE -DWORDINDEX_WITH_4_BYTE -c Perplexity.cpp -o optimized/Perplexity.o
[exec]
[exec] from Parameter.cpp:23:
[exec] make[1]: Entering directory '/home/ssetia/dr/joshua/src/giza-pp/mkcls-v2'
[exec] /usr/include/c++/4.9/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.g++ -Wall -W -DNDEBUG -O3 -funroll-loops -c GDAOptimization.cpp -o GDAOptimization.o
[exec]
[exec] #error This file requires compiler and library support for the
[exec] ^
[exec] In file included from /usr/include/c++/4.9/unordered_map:35:0,
[exec] from mystl.h:30,
[exec] from StatVar.h:32,
[exec] from Problem.h:34,
[exec] from Optimization.h:33,
[exec] from IterOptimization.h:33,
[exec] from GDAOptimization.h:31,
[exec] from GDAOptimization.cpp:27:
[exec] /usr/include/c++/4.9/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
[exec] #error This file requires compiler and library support for the
[exec] ^
[exec] In file included from /usr/include/c++/4.9/unordered_map:35:0,
[exec] from mystl.h:22,
[exec] from Vector.h:29,
[exec] from Perplexity.h:39,
[exec] from Perplexity.cpp:34:
[exec] /usr/include/c++/4.9/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
[exec] #error This file requires compiler and library support for the
[exec] ^
[exec] Makefile:92: recipe for target 'optimized/myassert.o' failed
[exec] g++ -Wall -W -DNDEBUG -O3 -funroll-loops -c HCOptimization.cpp -o HCOptimization.o
[exec] Makefile:11: recipe for target 'GDAOptimization.o' failed
[exec] Makefile:92: recipe for target 'optimized/Perplexity.o' failed
[exec] In file included from myassert.cpp:1:0:
[exec] mystl.h:170:30: error: expected template-name before ‘<’ token
[exec] #define MY_HASH_BASE hash_map<A,B,my_hash >
[exec] ^
[exec] mystl.h:173:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] mystl.h:170:30: error: expected ‘{’ before ‘<’ token
[exec] #define MY_HASH_BASE hash_map<A,B,my_hash >
[exec] ^
[exec] mystl.h:173:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] mystl.h:170:30: error: expected unqualified-id before ‘<’ token
[exec] #define MY_HASH_BASE hash_map<A,B,my_hash >
[exec] ^
[exec] mystl.h:173:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] make[1]: *** [optimized/myassert.o] Error 1
[exec] make[1]: *** Waiting for unfinished jobs....
[exec] In file included from Parameter.h:26:0,
[exec] from Parameter.cpp:23:
[exec] mystl.h:170:30: error: expected template-name before ‘<’ token
[exec] #define MY_HASH_BASE hash_map<A,B,my_hash >
[exec] ^
[exec] mystl.h:173:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] mystl.h:170:30: error: expected ‘{’ before ‘<’ token
[exec] #define MY_HASH_BASE hash_map<A,B,my_hash >
[exec] ^
[exec] mystl.h:173:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] mystl.h:170:30: error: expected unqualified-id before ‘<’ token
[exec] #define MY_HASH_BASE hash_map<A,B,my_hash >
[exec] ^
[exec] mystl.h:173:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] In file included from StatVar.h:32:0,
[exec] from Problem.h:34,
[exec] from Optimization.h:33,
[exec] from IterOptimization.h:33,
[exec] from GDAOptimization.h:31,
[exec] from GDAOptimization.cpp:27:
[exec] mystl.h:37:10: error: ‘hash’ is not a class template
[exec] struct hash<pair<T, V> > {
[exec] ^
[exec] mystl.h: In static member function ‘static void std::hash<T, V>::hash_combine(std::size_t&, const T&)’:
[exec] mystl.h:39:14: error: wrong number of template arguments (1, should be 2)
[exec] hash hasher;
[exec] ^
[exec] mystl.h:37:26: error: provided for ‘template<class T, class V> struct std::hash’
[exec] struct hash<pair<T, V> > {
[exec] ^
[exec] mystl.h:39:22: error: invalid type in declaration before ‘;’ token
[exec] hash hasher;
[exec] ^
[exec] In file included from Vector.h:29:0,
[exec] from Perplexity.h:39,
[exec] from Perplexity.cpp:34:
[exec] mystl.h:170:30: error: expected template-name before ‘<’ token
[exec] #define MY_HASH_BASE hash_map<A,B,my_hash >
[exec] ^
[exec] mystl.h:173:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] mystl.h:170:30: error: expected ‘{’ before ‘<’ token
[exec] #define MY_HASH_BASE hash_map<A,B,my_hash >
[exec] ^
[exec] mystl.h:173:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] mystl.h:170:30: error: expected unqualified-id before ‘<’ token
[exec] #define MY_HASH_BASE hash_map<A,B,my_hash >
[exec] ^
[exec] mystl.h:173:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] In file included from StatVar.h:33:0,
[exec] from Problem.h:34,
[exec] from Optimization.h:33,
[exec] from IterOptimization.h:33,
[exec] from GDAOptimization.h:31,
[exec] from GDAOptimization.cpp:27:
[exec] myleda.h: At global scope:
[exec] myleda.h:113:40: error: expected template-name before ‘<’ token
[exec] #define MY_HASH_BASE std::unordered_map<A,B>
[exec] ^
[exec] myleda.h:116:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] myleda.h:113:40: error: expected ‘{’ before ‘<’ token
[exec] #define MY_HASH_BASE std::unordered_map<A,B>
[exec] ^
[exec] myleda.h:116:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] myleda.h:113:40: error: expected unqualified-id before ‘<’ token
[exec] #define MY_HASH_BASE std::unordered_map<A,B>
[exec] ^
[exec] myleda.h:116:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] In file included from /usr/include/c++/4.9/unordered_map:35:0,
[exec] from mystl.h:30,
[exec] from StatVar.h:32,
[exec] from Problem.h:34,
[exec] from Optimization.h:33,
[exec] from IterOptimization.h:33,
[exec] from HCOptimization.h:31,
[exec] from HCOptimization.cpp:27:
[exec] /usr/include/c++/4.9/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
[exec] #error This file requires compiler and library support for the
[exec] ^
[exec] make[1]: *** [GDAOptimization.o] Error 1
[exec] make[1]: *** Waiting for unfinished jobs....
[exec] make[1]: *** [optimized/Perplexity.o] Error 1
[exec] Parameter.cpp: In function ‘bool writeParameters(std::ofstream&, const ParSet&, int)’:
[exec] Parameter.cpp:48:25: warning: ignoring return value of ‘char* getcwd(char_, size_t)’, declared with attribute warn_unused_result [-Wunused-result]
[exec] getcwd(path,1024);
[exec] ^
[exec] Makefile:92: recipe for target 'optimized/Parameter.o' failed
[exec] make[1]: Leaving directory '/home/ssetia/dr/joshua/src/giza-pp/GIZA++-v2'
[exec] Makefile:7: recipe for target 'gizapp' failed
[exec] Makefile:11: recipe for target 'HCOptimization.o' failed
[exec] make[1]: Leaving directory '/home/ssetia/dr/joshua/src/giza-pp/mkcls-v2'
[exec] Makefile:11: recipe for target 'mkcls-v2' failed
[exec] make: Leaving directory '/home/ssetia/dr/joshua/src/giza-pp'
[exec] make[1]: *_* [optimized/Parameter.o] Error 1
[exec] make: *** [gizapp] Error 2
[exec] make: *** Waiting for unfinished jobs....
[exec] In file included from StatVar.h:32:0,
[exec] from Problem.h:34,
[exec] from Optimization.h:33,
[exec] from IterOptimization.h:33,
[exec] from HCOptimization.h:31,
[exec] from HCOptimization.cpp:27:
[exec] mystl.h:37:10: error: ‘hash’ is not a class template
[exec] struct hash<pair<T, V> > {
[exec] ^
[exec] mystl.h: In static member function ‘static void std::hash<T, V>::hash_combine(std::size_t&, const T&)’:
[exec] mystl.h:39:14: error: wrong number of template arguments (1, should be 2)
[exec] hash hasher;
[exec] ^
[exec] mystl.h:37:26: error: provided for ‘template<class T, class V> struct std::hash’
[exec] struct hash<pair<T, V> > {
[exec] ^
[exec] mystl.h:39:22: error: invalid type in declaration before ‘;’ token
[exec] hash hasher;
[exec] ^
[exec] In file included from StatVar.h:33:0,
[exec] from Problem.h:34,
[exec] from Optimization.h:33,
[exec] from IterOptimization.h:33,
[exec] from HCOptimization.h:31,
[exec] from HCOptimization.cpp:27:
[exec] myleda.h: At global scope:
[exec] myleda.h:113:40: error: expected template-name before ‘<’ token
[exec] #define MY_HASH_BASE std::unordered_map<A,B>
[exec] ^
[exec] myleda.h:116:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] myleda.h:113:40: error: expected ‘{’ before ‘<’ token
[exec] #define MY_HASH_BASE std::unordered_map<A,B>
[exec] ^
[exec] myleda.h:116:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] myleda.h:113:40: error: expected unqualified-id before ‘<’ token
[exec] #define MY_HASH_BASE std::unordered_map<A,B>
[exec] ^
[exec] myleda.h:116:29: note: in expansion of macro ‘MY_HASH_BASE’
[exec] class leda_h_array : public MY_HASH_BASE
[exec] ^
[exec] make[1]: *** [HCOptimization.o] Error 1
[exec] make: *** [mkcls-v2] Error 2
[exec] Result: 2
[exec] make: Entering directory '/home/ssetia/dr/joshua/src/symal'
[exec] make: Nothing to be done for 'all'.
[exec] make: Leaving directory '/home/ssetia/dr/joshua/src/symal'

Do I need to change the make command to include the compiler options "-std=gnu++11"?

[mjpost]

Thanks. What did you set the environment variable $CXX to?

On Oct 31, 2015, at 1:58 AM, sidhantsetia [email protected] wrote:

@mjpost https://github.com/mjpost
I did not make any changes to $JOSHUA/jni/build_kenlm.sh. I declared an environment variable for CXX flags.
I also made a change to , where I added $JOSHUA/src/kenlm/CMakeLists.txt
set(CMAKE_CXX_FLAGS "-fPIC")

—
Reply to this email directly or view it on GitHub #226 (comment).

[mjpost]

I recommend just compiling it by yourself, and then putting the binaries in $JOSHUA/bin (GIZA++, snt2cooc.out, mkcls). I'm probably going to remove GIZA++ from Joshua because it's too much of a pain to keep C++ programs compiling.

[mjpost]

This has been fixed in the latest push.

[sidhantsetia]

Hi @mjpost

GIZA++ builds successfully now.
Now I'm getting a different error when I try to train the Indian model.

[kenlm] rebuilding...
dep=/home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en.uniq [CHANGED]
dep=lm.gz [NOT FOUND]
cmd=/home/ssetia/dr/joshua/bin/lmplz -o 5 -T /tmp -S 2G --verbose_header --text /home/ssetia/models/bn-en/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en.uniq | gzip -9n > lm.gz
took 0 seconds (0s)
[compile-kenlm] rebuilding...
dep=lm.gz [CHANGED]
dep=lm.kenlm [NOT FOUND]
cmd=/home/ssetia/dr/joshua/bin/build_binary lm.gz lm.kenlm
JOB FAILED (return code 1)
Reading lm.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

---

/home/ssetia/dr/joshua/src/kenlm/util/read_compressed.cc:384 in util::ReadBase* util::{anonymous}::ReadFactory(int, uint64_t&, const void*, std::size_t, bool) threw CompressedException.
This looks like a gzip file but gzip support was not compiled in. in file lm.gz
ERROR

How do I add gzip support?

[mjpost]

When you built KenLM, you must have removed the following flags, which are needed to compile in gzip support (these are taken from jni/build_kenlm.sh):

export CXXFLAGS+=" -O3 -fPIC -DHAVE_ZLIB"
export LDFLAGS+=" -lz"

[sidhantsetia]

These flags are still present in jni/build_kenlm.sh. I had included an environment variable CMAKE_CXX_FLAGS="-fPIC". If I change the value of this variable kenlm does not build and throws an error.

[mjpost]

Somehow your programs didn't compile against libz. I suggest wiping out bin/{build_binary,lmplz,query} and src/kenlm/build, and rebuilding. Make sure you have the development versions of the libz libraries installed. This should be standard.

[sidhantsetia]

I did all of the above but I'm still getting the same error. I also removed the environment variable CMAKE_CXX_FLAGS.
The only change I have from the latest source code is that I have added set(CMAKE_CXX_FLAGS "-fPIC") to the file $JOSHUA/src/kenlm/CMakeLists.txt. If I remove this line then I'm unable to build kenlm.

[mjpost]

I'm sorry, I can't help. I suggest you try to compile the code directly from KenLM, then try to run build_binary manually. If that doesn't work, you'll know it's a problem on your system.