Skip to content

CentOS Build Instructions

Jump to bottom
Andrew R. Lake edited this page Mar 18, 2015 · 11 revisions

Introduction

This page contains information about building various components of the Toolkit in CentOS 6.

Kernel

The Toolkit includes a custom version of the CentOS 6 RHEL (Red Hat Enterprise Linux) kernel.

To build this kernel, you will need a host with CentOS 6, and the 'rpm-build' package installed (optionally mock) with at least 15 GB of space available.

Preparing The Kernel Source

Building The Kernel

  • To build the kernel SRPM run: rpmbuild --define "_sourcedir ." --define "_srcrpmdir ." --nodeps -bs *spec
  • If the above commands executed properly you should see a kernel SRPM file
    • If not look through the output to determine what went wrong
  • To build the kernel run mock (make sure your user is in the "mock" group):
    • x86_64: mock -r epel-6-x86_64 --resultdir=$HOME/mock-results/epel-6-x86_64/ --arch=x86_64 --without=debug --without=kabichk kernel_file.src.rpm
    • i386: mock -r epel-6-i386 --resultdir=$HOME/mock-results/epel-6-i386 --arch=i386 --target=i686 --without=debug --without=kabichk kernel_file.src.rpm
    • You also need to run a third command to build the firmware as noarch:
      • mock -r epel-6-i386 --resultdir=$HOME/mock-results/epel-6-i386 --arch=i386 --target=noarch --without=debug --without=kabichk --with=firmware kernel_file.src.rpm
    • If new patches have been added, these commands may error out due to patches that don't apply cleanly. If so, these patches will need modified as appropriate and the command rerun (look here)

Kernel Modules aufs and sk98lin

NOTE: We don't need these anymore. leaving for now for posterity

Every release requires you to rebuild the aufs and sk98lin modules as well as the kernel. The process for both is as follows.

rpm -iv aufs-kmod-file.src.rpm
rpm -iv sk98lin-kmod-file.src.rpm
  • Update the installed SPEC file (in ~/rpmbuild/SPECS) with the following changes:
    • Update the kversion to match the kernel you are building against
    • Increase the Release by 1
  • Rebuild the source RPM using the new SPEC file, which resides in ~/rpmbuild/SPECS
rpmbuild -bs SPECS/aufs-kmod.spec
rpmbuild -bs SPECS/sk98lin-kmod.spec
  • If using mock, copy the kernel rpms to your holding repo (see below) and then start the build. This should complete the build process.
    • mock -r epel-6-x86_64 --resultdir=$HOME/mock-results/epel-6-x86_64/ --arch=x86_64 --without=debug --without=kabichk aufs_file.src.rpm
    • mock -r epel-6-x86_64 --resultdir=$HOME/mock-results/epel-6-x86_64/ --arch=x86_64 --without=debug --without=kabichk sk98_file.src.rpm
    • mock -r epel-6-i386 --resultdir=$HOME/mock-results/epel-6-i386/ --arch=i386 --target=i686 --without=debug aufs_file.src.rpm
    • mock -r epel-6-i386 --resultdir=$HOME/mock-results/epel-6-i386/ --arch=i386 --target=i686 --without=debug --without=kabichk sk98_file.src.rpm

Setting up a holding repo

In /etc/mock/epel-6-i386.cfg:

holding name=holding baseurl=file:///home/mock/holding/epel-6/i386/

Create a corresponding entry in /etc/mock/epel-6-x86_64.cfg.

The directories /home/mock/holding/[epel-5|epel-6]/[i386|x86_64] are yum repositories created using createrepo. You can copy the kernel RPMs in, and then do this:

createrepo -u /home/mock/holding/epel-6/i386 createrepo -u /home/mock/holding/epel-6/x86_64

For more details about mock, see this page.

Generating Random Bytes

When compiling on a remote host, it is common to see an error like this:

Not enough random bytes available.  Please do some other work to give
the OS a chance to collect more entropy! (Need 266 more bytes)

There are a number of ways to generate random bytes on a remote machine, but the easiest way that we've found is to perform parallel 'hdparm' tests. The following script can be run one or more times in the event that the error above is seen.

#!/bin/bash

NUM_TESTS=20

if [ -n "$1" ]; then
    NUM_TESTS=$1
fi

for i in `seq 1 $NUM_TESTS`; do
    /sbin/hdparm -t /dev/sda1 &
done

Creating Kernel Patches

Unfortunately all of the kernel patches the Toolkit utilizes are intended for the mainstream kernel releases and not the RHEL version that CentOS 6 uses. The RHEL kernel often backports fixes from newer kernel versions essentially creating a hybrid. This presents many issues when applying a patch to the RHEL kernel.

Most patches must first be applied manually (by hand) and then regenerated by running a diff between the vanilla RHEL kernel and the patched one: eg. diff -ur kernel.old/ kernel.new/ > kernel.patch. In the case of the Toolkit we incorporate two custom patches to the kernel described above: aufs and web100. Likely within the same major kernel version (ie. 2.6.32) the existing patches will apply successfully but when a new kernel is released they may have to be recreated or modified.

AuFS

Aufs represents a unique problem with using the RHEL kernel. Many of the methods it utilizes are rewritten between the mainstream and the RHEL kernel (the RHEL kernel often includes updates applied to later versions of the mainstream kernel). For this reason, creating a successful and stable aufs patch is quite difficult. To minimize the impacts of patching the kernel there are two patches involved: one patch contains completely necessary and unavoidable patches that must be applied to the kernel, and the other contains patches for the aufs kernel module itself to make it work with the RHEL kernel. Both patches must be created with extreme care as there must be integrity maintained between the kernel, the aufs kernel module, and the aufs user utilites.

To create a new aufs patch will require some understanding of how the kernel operates and experience in programming, as parts of it will likely have to be modified or rewritten by hand. Because we want to use the CentOS RHEL kernel we must use the aufs standalone branch and not the one with full kernel sources. Aufs utilities are also necessary to use aufs but those are a separate package and for the most part can be ignored throughout this process. To patch a new kernel, it is recommended to download several versions of aufs (eg. if your kernel version is 2.6.32 then download aufs intended for mainline kernel versions 2.6.30 through 2.6.36).

Aufs standalone comes with several patches, however the only necessary ones are: aufs2-base.patch, aufs2-standalone.patch, and proc_map.patch. Try patching the CentOS RHEL kernel with these three patches from all the aufs versions you downloaded, each in turn. Determine which group of patches applies the cleanest (has the least errors). Do not delete the extra aufs versions as they may be used later. Start applying the cleanest patches by hand, and if there are issues or references to missing blocks of code consult with the other versions of aufs while taking careful note of what was changed (you may wish to download the full kernel source files from aufs to accomplish this).

After this you are ready to build the kernel using the steps listed above. If the kernel fails to build consult the build logs to find the issue(s).

The final step in this process is making the aufs kernel module. The initial kernel patch may need to be modified to work with aufs and most likely the aufs sources will have to be patched as well. Try building the kernel module against your patched kernel sources and read though the output logs to address any errors. Likely this process will have to be repeated several times until all the issues are resolved. This part is by far the most difficult, so consult the existing aufs patches for older CentOS RHEL kernel versions.

When this is done you should have a working aufs compatible RHEL kernel and the aufs kernel module.

Aufs kernel patches for various kernel versions can be found here.

Web100

Fortunately the web100 kernel patch seems to be much a much smoother process as it requires less modification. Manual patching is highly recommended to make sure that things end up where they need to be. Through experience, I have found that lines of code can be inserted where they don't belong which will cause major issues when building and using a kernel. After manual patching is completed simply run a diff between the vanilla kernel and the modified one to produce a patch that will be applied at RPM build time.

Web100 kernel patches for various kernel versions can be found here.

Customizing The Toolkit

The pS-Performance Toolkit, as the LiveCD, LiveUSB, and NetInstall, are built using custom CentOS kickstart files. You can read more about kickstart installations here. The kickstart file defines the set of software repositories to use, the set of software to install, and a script that performs some system configuration once the software has been installed. Customizing the toolkit will involve changing one or more of those elements in the kickstart files.

Pre-Requisites

The build procedures here assume that the toolkit ISOs are being built on CentOS 6. The steps may work on other systems, but have only been tested on CentOS 6.

For building custom versions of the LiveCD, LiveUSB, or NetInstall, you'll need a copy of ImageMagick installed on the host. This should be available via yum.

For building custom versions of the NetInstall, you will need a copy of mkisofs installed on the host. This should be available via yum.

For building custom versions of the LiveCD and LiveUSB, you will need the livecd-tools package installed on CentOS 6. This should also be available via yum. You can find more installation information here.

Building The Toolkit

You can download the kickstarts, images and scripts needed to build the toolkit by doing a git checkout of http://perfsonar-ps.googlecode.com/git/Toolkit_Building/

There are 6 directories in this git repository. The ones of importance for customization are kickstarts, images, and scripts. The kickstarts directory contains the files used for generating the kickstarts used by the LiveCD and NetInstall setups. The scripts directory contains 3 scripts used for building the toolkit, one for the LiveCD, one for the LiveUSB, and one for the NetInstall as well as a script used to deploy the LiveUSB. The images directory contains the image displayed when the CD is first booted.

If you run the script build_livecd.sh or build_liveusb.sh as root, it should download all the necessary RPMs, and build an ISO of the LiveCD/USB and place it in the resources directory. Similarly, if you run the script build_netinstall.sh as root, it should build an ISO of the NetInstall.

As of the CentOS 6 release, there is an option to specify whether to build a 32-bit or 64-bit version. If the script is executed without any parameters it will automatically build a 32-bit version of the Toolkit, however if you wish to build a 64-bit version you must supply the script with an architecture parameter. For example, to build a 64-bit version of the LiveCD you would run build_livecd.sh --arch x86_64 as root. Note that this requires all Toolkit software to be rebuilt for 64-bit compatibility.

The general build process of the LiveCD/USB proceeds as follows:

  • Patches the 'generic' kickstart file (centos6-base.cfg) with LiveCD-specific modifications (centos6-livecd/usb.patch)
  • Calls livecd-creator to build the LiveCD/USB from the kickstart file
  • Modifies the generated LiveCD/USB to use the proper boot image

The general build process of the NetInstall roceeds as follows:

  • Patches the 'generic' kickstart file (centos6-base.cfg) with NetInstall-specific modifications (centos6-netinstall.patch)
  • Modifies the generic CentOS netinstall CD (in the resources directory) to include the kickstart and to boot using it
  • Modifies the generic CentOS netinstall CD to use the proper boot image

Customizing The Toolkit

To customize the toolkit, you will need to modify the kickstart file. The centos6-base.cfg file in the kickstarts directory is the 'generic' kickstart file. There are 3 patches: centos6-livecd.patch, centos6-liveusb.patch, and centos6-netinstall.patch, that modify this kickstart file.

There are a large number of directives inside the kickstart. The meaning of most of these is outside the scope of this document, but should be available here.

Customizing The Software Yum Repositories

The set of repositories that will be used to retrieve RPMs from can be customized. This allows you to use a local mirror to speed up builds as well as to include new or customized RPMs. Note, however, that when changing the repositories, you must make sure that the software to be installed (either directly or to satisfy dependencies) must still be available in the set of repositories.

The 'repo' lines specify the set of repositories to use:

repo --name=a-Internet2    --baseurl=http://software.internet2.edu/rpms/i386/main/

The above line tells the kickstart to use the Internet2 repository and gives the URL of the yum repository.

Note: the repositories here are only used for installation. They will not be available after installation. If you want to make these software repositories available after installation, you must either have the kickstart install an appropriate repository package (see here for an example of the repository package), or have the post installation script add the appropriate .repo and key files.

Customizing The Software To Install

In the %packages section, there are a list of packages to install or not install. Any packages that start with "@" are package groups (e.g. @core is the set of core packages). Any packages that start with "-" are packages that should not be installed (e.g. -sysreport says to not install the sysreport package). All other packages are normal packages to install like you might do with yum.

You can add any packages here that you might want installed.

Customizing The Toolkit System Environment

The toolkit system environment can be configured using the %post script. This script is run after the software packages have been installed, and allow for modifications to be made to the installed system before it is booted for the first time (or, in the case of the LiveCD/USB, before it is turned into an unmodifiable ISO).

There is an existing %post script that performs a number of tasks needed by the toolkit. However, this script can be modified to include any desired configuration for the toolkit system. For example, users could be added, or configuration files could be modified. The existing post installation script includes a wide variety of methods for configuring aspects of the installed system.

The thing to remember when making changes is that the patch files expect to patch the centos6-base.cfg file. There are a few ways to go about making changes. One way is to make changes to the centos6-base.cfg in such a way that they don't conflict with the patches. This is generally the path of least resistance. Another option, if you only plan on making one of the LiveCD/USBs or NetInstalls, is to patch the centos6-base.cfg before making your changes, and then edit the build_livecd.sh, build_liveusb.sh, or build_netinstall.sh and set the KICKSTART_PATCH=... line to be KICKSTART_PATCH=. This will stop the script from trying to patch the centos6-base.cfg file. Lastly, you can edit the build_netinstall.sh, and the regenerate or edit the patch files to account for the changes.

Building Toolkit Packages With Mock

This page contains background information on mock. The latter sections assume basic knowledge of mock and 'holding' repositories, and that the holding repository has been configured as described.

This page contains information on creating and building source RPMs. The latter sections assume basic knowledge of building source RPMs.

i2_mock utility

To ease the use of mock, we built a simple wrapper script over the mock command which simply sets some default values:

  • The uniqueext value will be set to the callers username
  • The architecture is set to "i386"
  • The default build environment is set to "epel-6" (RedHat6 + the EPEL repositories)
  • The output directory is set to /home/username/holding/environment/architecture/name 
#!/usr/bin/perl

use strict;
use warnings;

use Getopt::Long;

Getopt::Long::Configure("pass_through");
Getopt::Long::Configure("no_ignore_case");

my $uniqueext;
my $resultdir;
my $arch;
my $distro;

my $result = GetOptions("uniqueext=s", \$uniqueext, "resultdir=s", \$resultdir, "arch=s", \$arch, "distro=s", \$distro);

my @remaining_args = @ARGV;

my $srpm = $ARGV[$#ARGV];
unless ($srpm)
{
	print "No source RPM specified.\n";
	exit(-1);
}

unless (`rpm -qp $srpm 2>/dev/null`)
{
	print "Invalid SRPM: $srpm";
	exit(-1);
}

my ($name, $pass, $uid, $gid, $quota, $comment, $gcos, $dir, $shell, $expire) = getpwuid($>);

$arch = "i386" unless ($arch);
$distro = "epel-6" unless ($distro);
$uniqueext = $name unless ($uniqueext);
$resultdir = "/home/$name/holding/$distro/$arch/" . `rpm -qp $srpm 2>/dev/null` unless ($resultdir);
chomp($resultdir);

my $mycfg = "$distro-$arch";
$mycfg = "$distro-i386" if ( ! -e "/etc/mock/$mycfg" and $arch eq "i686" and -e "/etc/mock/$distro-i386.cfg");

my @args = ();
push @args, ("/usr/bin/mock");
push @args, ("-r", $mycfg);
push @args, ("--resultdir", $resultdir);
push @args, ("--uniqueext", $uniqueext);
push @args, ("--arch", $arch);
push @args, @remaining_args;

print join(" ", @args) . "\n";
system(@args);

I2utils

web100_userland

bwctl

owamp

ndt

  • Place a built RPM of I2utils into the 'holding' repository
  • Place a built RPM of web100_userland into the 'holding' repository
  • Build a source RPM of ndt (e.g. ndt-3.6.4-3.el6.src.rpm)
  • Run: i2_mock [ndt SRPM] (e.g. ndt-3.6.4-3.el6.src.rpm)

npad

nuttcp

dbxml

perl dbxml

  • Place a built RPM of dbxml into the 'holding' repository
  • Build a source RPM of perl-dbxml (e.g. perl-dbxml-2.5.16-1.el6.src.rpm)
  • Run: i2_mock [perl-dbxml SRPM] (e.g. i2_mock perl-dbxml-2.5.16-1.el6.src.rpm)

kernel

  • Build a source RPM of the kernel. This will involve steps described above. (e.g. kernel-2.6.32-279.2.1.el6.aufs.web100.src.rpm)
  • Run i2_mock --arch i686 --without debug --without kabichk [kernel SRPM] (e.g. i2_mock --arch i686 --without debug --without kabichk kernel-2.6.32-279.2.1.el6.aufs.web100.src.rpm)
    • NOTE: make sure you have at least 5 GB of available disk space for building the set of kernels

aufs kernel module

  • Copy the kernel development RPMS (kernel-devel) into the 'holding' repository
    • Make note of the kernel version number (e.g. 2.6.32-279.2.1.el6.aufs.web100)
  • Build a source RPM of the aufs module (e.g. aufs-kmod-2.2-1.el6.src.rpm)
  • Run i2_mock --arch i686 -D "kversion [kernel version number]" [aufs-kmod SRPM] (e.g. i2_mock --arch i686 -D "kversion 2.6.32-279.2.1.el6.aufs.web100" aufs-kmod-2.2-1.el6.src.rpm)

aufs utilities

sk98lin kernel module

  • Copy the kernel development RPMS (kernel-devel) into the 'holding' repository
    • Make note of the kernel version number (e.g. 2.6.32-279.2.1.el6.aufs.web100)
  • Build a source RPM of the sk98lin module (e.g. sk98lin-kmod-10.92.1.3-1.el6.src.rpm)
  • Run i2_mock --arch i686 -D "kversion [kernel version number]" [sk98lin-kmod SRPM] (e.g. i2_mock --arch i686 -D "kversion 2.6.32-279.2.1.el6.aufs.web100" sk98lin-kmod-10.92.1.3-1.el6.src.rpm)

Useful Links

Reporting an Issue

Project Planning

Support Rotation

Writing Code

Release Management

Clone this wiki locally