Implement extractor for builder metadata

[jsheunis]

This would mainly extract information from the singularity recipe, specifically the "for"-distribution. This information is implicitly necessary on the distribution level, because it defines what the distribution is for. Still to be figured out and decided if this information should then be aggregated to the distribution dataset level in order for it to be represented there...

[jsheunis]

Code/comments from exploring https://deb.datalad.org/ with spython (singularity-cli):

Install distribution dataset:

datalad clone ria+https://deb.datalad.org/store#~dist-bullseye+datalad
cd dist-bullseye+datalad
datalad get -d . -r -R 1 --no-data

Install spython:

git clone https://www.github.com/singularityhub/singularity-cli.git
cd singularity-cli
python setup.py install

Parse singularity recipe file: dist-bullseye+datalad/builder/recipes/singularity-any with some code, and print out variables:

from spython.main.parse.parsers import SingularityParser
recipefile = 'dist-bullseye+datalad/builder/recipes/singularity-any'
parser = SingularityParser(recipefile)
from pprint import pprint
pprint(vars(parser.recipe['spython-base']))

Output:

WARNING SETUP is error prone, please check output.
WARNING More than one line detected for runscript!
WARNING These will be echoed into a single script to call.
{'cmd': '/bin/bash /entrypoint.sh',
 'comments': ["#  ingest any initialization files deposited in 'init/'",
              '#  (intentionally not using `cp -L` to be able to',
              '#   deposit symlinks)',
              '# !/bin/bash'],
 'entrypoint': None,
 'environ': [],
 'files': [],
 'fromHeader': 'debian:bullseye',
 'install': ["sed -i 's,main,main,g' /etc/apt/sources.list",
             'apt-get -y update',
             'apt-get -y install --no-install-recommends build-essential '
             'devscripts eatmydata equivs moreutils lintian',
             '# remove everything but the lock file from',
             '# /var/cache/apt/archives/ and /var/cache/apt/archives/partial/',
             'apt-get clean',
             '# builder setup relies on this particular directory to exist '
             'inside the container',
             'mkdir /pkg',
             'cat << EOT > /doall.sh',
             'set -e -u',
             'dsc=\\$1',
             '# somehow --containall causes 755 by default',
             'chmod 777 /tmp',
             '# all building will take place inside the container',
             '# maybe add option to do it on the host FS?',
             'mkdir -p /tmp/build',
             '# ingest the source package into a defined location in the '
             'container env',
             'dcmd cp "\\$dsc" /tmp/build',
             '# extract the source package',
             'echo -e "\\n#\\n# Extracting the source package: \\$(date -u '
             '--iso-8601=seconds)\\n#\\n"',
             '(cd /tmp/build && dpkg-source -x *.dsc source)',
             '# update the package list, necessary to get the latest '
             'build-dependency',
             '# version',
             'echo -e "\\n#\\n# Updating build environment: \\$(date -u '
             '--iso-8601=seconds)\\n#\\n"',
             'chronic eatmydata apt-get update -y',
             '# update the base environment. needed for rolling releases to '
             'stay',
             '# up-to-date',
             'chronic eatmydata apt-get upgrade -y',
             '# install the declared build-dependencies',
             'echo -e "\\n#\\n# Installing build-dependencies: \\$(date -u '
             '--iso-8601=seconds)\\n#\\n"',
             "(cd /tmp && mk-build-deps -t 'eatmydata apt-get -o "
             "Debug::pkgProblemResolver=yes --no-install-recommends -y' -i -r "
             '/tmp/build/source/debian/control)',
             '# build the binary package(s)',
             'echo -e "\\n#\\n# Build starting: \\$(date -u '
             '--iso-8601=seconds)\\n#\\n"',
             "# we need lintian's allow-root, because we are running as root "
             "inside singularity's",
             '# fakeroot environment',
             '(cd /tmp/build/source && debuild -uc -us -b --lintian-opts '
             '--allow-root)',
             '# deposit the results',
             'echo -e "\\n#\\n# Deposit build results: \\$(date -u '
             '--iso-8601=seconds)\\n#\\n"',
             'dcmd cp /tmp/build/*changes /pkg',
             'EOT',
             '# look for any finalizer executables and run them',
             '[ -d /finalize ] && (for finalizer in /finalize/*; do '
             '".${finalizer}"; done) || true',
             'echo "set -e -u" >> /entrypoint.sh',
             'echo "# singularity will (likely) try to provide the hosts '
             'timezone in the container" >> /entrypoint.sh',
             'echo "# but this would prevent certain upgrade scenarios (e.g., '
             'a tzdata update), so" >> /entrypoint.sh',
             'echo "# undo this" >> /entrypoint.sh',
             'echo "umount /usr/share/zoneinfo/Etc/UTC || true" >> '
             '/entrypoint.sh',
             'echo "dsc=$1" >> /entrypoint.sh',
             'echo "# ISO-like timestamp, but more compact" >> /entrypoint.sh',
             'echo "ts=$(date --utc --iso-8601=seconds | cut -d \'+\' -f 1 | '
             'tr -d \':-\')" >> /entrypoint.sh',
             'echo "flavor=$(dpkg-architecture -q DEB_BUILD_ARCH_CPU)" >> '
             '/entrypoint.sh',
             'echo "# double-braces are escaping for template placeholder '
             'expansion" >> /entrypoint.sh',
             'echo "logbase="${dsc%*.dsc}"" >> /entrypoint.sh',
             'echo "logfile="/pkg/logs/${logbase}_${ts}_${flavor}.txt"" >> '
             '/entrypoint.sh',
             'echo "mkdir -p /pkg/logs" >> /entrypoint.sh',
             'echo "bash /doall.sh "$dsc" |& tee "$logfile"" >> /entrypoint.sh',
             'echo "echo -e "\\n#\\n# Builder exit: $(date -u '
             '--iso-8601=seconds)\\n#\\n" >> "$logfile"" >> /entrypoint.sh',
             'chmod u+x /entrypoint.sh'],
 'labels': [],
 'layer': 1,
 'layer_files': {},
 'ports': [],
 'source': '/Users/jsheunis/Documents/psyinf/dist-bullseye+datalad/builder/recipes/singularity-any',
 'test': None,
 'volumes': [],
 'workdir': None}