Make process improvements

[gouenji-shuuya]

• Move to bash as that was implicitly expected (ref. #vidyut on Discord)
• Some refactoring.
• Sub-make is correctly called when using make create_all_data.
• Use -jnproc in make.
• Ignore venv in git.

[akprasad]

Thank you for this! Forgive the late look.

[akprasad]

Thank you very much!

[akprasad]

My understanding is that -j controls the number of make recipes that run in parallel. If so, what is the benefit of using -j here?

[gouenji-shuuya]

Cargo can respect the jobserver settings. Refer rust-lang/rust#42682.

Even if the workflow is currently serial, there is a possibility of decoupling steps in future for faster builds.

For instance, create_kosha executes successfully before create_sandhi_rules, so there is no dependency (though the latter is quick). I think the cloning can also be parallelized, by use of recipes like this:

mkdir build

get_corpus_data:
	@if [[ -e "data/raw/dcs" ]]; then 				\
		echo "Training data already exists -- skipping fetch.";	\
	else 								\
		echo "Training data does not exist -- fetching.";	\
		mkdir -p "data/raw/dcs";				\
		git clone https://github.com/OliverHellwig/sanskrit.git	\
			--depth=1 build/dcs-data;			\
		mv build/dcs-data/dcs/data/conllu data/raw/dcs/conllu;	\
	fi


get_linguistic_data:
	@if [[ -e "data/raw/lex" ]]; then 				\
		echo "Lexical data already exists -- skipping fetch.";	\
	else 								\
		echo "Lexical data does not exist -- fetching.";	\
		mkdir -p "data/raw/lex";				\
		git clone https://github.com/sanskrit/data.git		\
			--depth=1 build/data-git;			\
		python3 build/data-git/bin/make_data.py 		\
			--make_prefixed_verbals;			\
		mv build/data-git/all-data/* data/raw/lex;		\
	fi

So the create_all_data.sh file really is a bottleneck. There are several make commands there and they execute one by one. Maybe a bit of parallelization will help, like in test/train?