Skip to content

Latest commit

 

History

History
313 lines (200 loc) · 7.45 KB

HACKING.md

File metadata and controls

313 lines (200 loc) · 7.45 KB

Hacking on GNU Datamash

This file contains random collection of hacking tid bits.

Building from git

Use the following commands to build the latest version of datamash (including development/unreleased features):

git clone git://git.savannnah.gnu.org/datamash.git
cd datamash
./bootstrap
./configure
make
make check

The following programs and versions are required when building from git sources:

autoconf (>=2.62)
automake (>=1.11.1)
autopoint and gettext (>=0.19.4)
gperf
pkg-config
makeinfo (part of GNU texinfo)
perl
wget

These programs are for development from git but are not needed when building from the released tar archives found on https://ftp.gnu.org/gnu/datamash/ .

Compilation Options

To use a specific compiler (e.g. clang), run:

./configure CC=clang

To enable warnings-as-errors (-Werror), run:

./configure --enable-werror

Warnings-as-errors is enabled automatically when building from git repository. To disable, run:

./configure --disable-werror

To disable code optimization (e.g. for easier debugging), run:

# during configuration
./configure CFLAGS="-g -O0"
# or during make
make CFLAGS="-g -O0"

To enable LINT mode:

./configure --enable-lint

To test Debian-hardening configuration, run:

make deb-hard

Testing

Common tests

To run the most common tests:

make check

To run a specific test:

make check TESTS=./tests/datamash-tests.pl

Expensive tests

Expensive tests use valgrind to find memory usage errors, and use specially mounted file-system to emulate errors. To create the special file systems, run:

./build-aux/create_corrupted_file_system.sh
mkdir /tmp/badfs/
sudo mount -o sync bad_disk.img /tmp/badfs/

./build-aux/create_small_file_system.sh
mkdir /tmp/fullfs/
sudo mount -o sync tiny_disk.img /tmp/fullfs/
sudo chown $USER /tmp/fullfs/

Then run the expensive tests:

make check-expensive

Debug information in unit tests

To enable debug information when testing (to debug the testing framework):

make check VERBOSE=yes DEBUG=1

The DEBUG environment variable is checked in ./tests/Coreutils.pm.

Test tarball distribution

To create a tarball (for a new release) and test it run:

make distcheck

The filename will be based on the current git status (e.g. datamash-1.0.6.40-8207-dirty.tar.gz).

Test on remote systems

To test a tarball (or a git repository) on a remote host, run:

./build-aux/check-remote-make.sh FILE.tar.gz HOST

The FILE.tar.gz will be copied (scp) to host HOST, then it will be compiled using ./configure && make && make check. HOST must be configured for password-less login.

See ./build-aux/check-remote-make.sh -h for many more options.

Coverage

To generate coverage information using gcov and lcov, run:

make coverage

Coverage information will be stored in:

./doc/coverage/index.html

By default, the coverage rule runs the non-expensive tests. To check coverage with expensive tests, run:

make coverage-expensive

It seems recent versions of lcov disabled branch coverage by default (checking only line and function coverage). To re-enable branch coverage, either:

  • edit your .lcovrc file (e.g /etc/lcovrc) and change lcov_branch_coverage setting to 1
  • add --rc lcov_branch_coverage=1 to your lcov command line

Static Code Analysis

Static Analysis target uses clang's scan-build tool. Run:

make static-analysis

Coding Style

To check the source code files against the most common syntax rules, run:

make syntax-check

Most syntax-check rules are defined in maint.mk (auto-generated from gnulib's ./gnulib/top/maint.mk). Few additional rules are defined in cfg.mk (copied from GNU Coreutils).

Syntax-check rules begin with sc_ (e.g. sc_long_lines). The rule name will appear when syntax-check fails:

$ make syntax-check
...
...
maint.mk: found trailing blank(s)
make: *** [sc_trailing_blank] Error 1

After fixing the offending file, to check this rule alone (instead of re-checking all syntax-check rules), use:

make sc_traling_blank

Running the script hooks/setup-hooks.sh will install a git pre-commit hook that automatically runs a syntax check.

Make tips

Silent build rules

To show detailed build commands, run:

# During configure
./configure --disable-silent-rules
# or during make
make V=1

To hide detailed build commands (showing only the file being built), run:

# During configure
./configure --enable-silent-rules
# or during make
make V=0

Non-Recursive Make

The build system uses non recursive makefile structure. Search for the 1997 paper "Recursive Make Considered Harmful" by Peter Miller, and see examples of switching to non-recursive make in GNU Coreutils, GNU Hello, GNU Datamash

Makefile structure

Makefile.am is the main makefile, which generates Makefile.in and finally Makefile during ./bootstrap or autoreconf -i.

Makefile.am includes the following items:

  • lists source files in datamash_SOURCES
  • lists tests in TESTS.
  • ./lib/local.mk - gnulib files
  • ./doc/local.mk - manual generation

The generated Makefile makes use of the following files:

  • maint.mk - maintenance targets, from gnulib.
  • cfg.mk - custom targets and configurations for datamash.

Portability

Windows using mingw

Compilation for windows using MinGW compilers requires the following CFLAGS:

CFLAGS="-D__USE_MINGW_ANSI_STDIO=1"

./configure supports an option --enable-mingw which sets up the correct flags. This option should be automatically detected correctly when using cross-compilation, such as:

./configure --host=x86_64-w64-mingw32

Compilation under Cygwin does not require this flag. See related discussion: https://lists.gnu.org/archive/html/bug-gnulib/2014-09/msg00052.html

OpenBSD

Bootstrap instructions for OpenBSD (Tested with 7.2).

Install prerequisites

Make sure the following packages are installed with pkg_add

  • autoconf 2.71
  • automake 1.16.5
  • gettext-tools
  • git
  • gperf
  • wget

Bootstrap, configure and compile

First, of course, get the source:

$ git clone git://git.sv.gnu.org/datamash
$ cd datamash

Then run the bootstrap script (This can take a while):

$ AUTOCONF_VERSION=2.71 AUTOMAKE_VERSION=1.16 ./bootstrap

Then configure and make like usual:

$ CC=clang ./configure --disable-werror
$ make
$ make check

NetBSD

Bootstrap instructions for NetBSD (Tested with 9.2).

Install prerequisites

Make sure the following packages are installed with pkgin or another pkgsrc interface:

  • autoconf
  • automake
  • ca-certificates
  • gettext
  • git
  • gmake
  • gperf
  • perl
  • pkgconf
  • wget

Bootstrap, configure and compile

First, of course, get the source:

$ git clone git://git.sv.gnu.org/datamash
$ cd datamash

Then run the bootstrap script (This can take a while):

$ PATH="/usr/pkg/bin:$PATH" ./bootstrap

(NetBSD has a /usr/bin/gettext that isn't compatible with the GNU gettext one installed as /usr/pkg/bin/gettext so the latter needs to come first in the path)

Then configure and make like usual:

$ ./configure --disable-werror
$ gmake
$ gmake check