Skip to content

Latest commit

 

History

History
152 lines (120 loc) · 6.23 KB

BUILD.md

File metadata and controls

152 lines (120 loc) · 6.23 KB

Build

Dependencies

The build process of OpenIO SDS depends on several third-party projects.

When building only the SDK, OpenIO only depends on:

  • cmake, make: involved in the build process.
  • bison, flex: generates expression parsers.
  • glib2, glib2-devel
  • curl, libcurl, libcurl-devel
  • json-c, json-c-devel
  • asn1c: Now only necessary at the compile time, this is our ASN.1 codec forked from Lev Walkin's excellent ASN.1 codec. The purpose of our fork is simply to provide codec for explicitely sized integers (int{8,16,32,64} instead of long int) and GLib-2.0 memory allocations. The forked version is required only when building code prior to version 6.0.0.

Building the entire project will require the SDK dependencies, but also:

  • python: Pure python code generator (no dependency), and python modules.
  • python-distutils-extra: required for the installation process
  • httpd, httpd-devel: server base for ECD service (and rawx for code prior to version 6.0.0)
  • apr, apr-util-devel, apr-devel: internally used by rawx modules (prior to version 6.0.0)
  • attr, libattr-devel: we use xattr a lot to stamp rawx chunks and repositories base directory.
  • sqlite, sqlite-devel: base storage for META{0,1,2} services.
  • zeromq3, zeromq3-devel: communication of events between services and forward agents.
  • zookeeper-devel, libzookeeper_mt.so: building with distribution's zookeeper client is OK, but the package ships with a lot of dependencies, including the openjdk. We recommand to use the official Oracle/Sun JDK, and to build your own zookeeper client from the source to avoid a huge waste of space and bandwith.
  • python-setuptools
  • python-pbr
  • beanstalkd: you need it to have the event-agent working
  • libapache2-mod-wsgi-py3 (as named on Ubuntu), the WSGI module for apache2

In addition, there some dependencies at runtime (the up-to-date list is in requirements.txt). You don't need to install them on the system, they will be installed by pip in your virtualenv (see Building).

  • python-eventlet
  • python-werkzeug
  • python-gunicorn
  • python-redis
  • python-requests
  • python-simplejson
  • python-cliff
  • python-pyeclib
  • python-futures

The account service will require an up and running backend:

  • FoundationDB

Generating the documentation will require:

  • epydoc: available in your python virtualenv

Configuration

The Makefile's generation is performed by cmake. The master CMake directives files accepts several options. Each option has to be specified on the cmake's command line with the following format:

cmake -D${K}=${V} ${SRCDIR}

In addition to common cmake options, these specific options are also available:

Directive Help
LD_LIBDIR Path suffix to the installation prefix, to define the default directory for libraries. E.g. "lib" or "lib64", depending on the architecture.
STACK_PROTECTOR Trigger stack protection code. Only active when CMAKE_BUILD_TYPE is set to "Debug" or "RelWithDebInfo"
ALLOW_BACKTRACE generate backtraces in errors.
FORBID_DEPRECATED define it to turn into errors the warnings for deprecated symbols from the GLib2.
EXE_PREFIX Defines a prefix to all CLI tool. By default, set to "sds".
SOCKET_OPTIMIZED define if to use socket3 and accept4 syscalls
SOCKET_DEFAULT_LINGER_ONOFF (integer value) triggers the onoff value of the SO_LINGER configuration.
SOCKET_DEFAULT_LINGER_DELAY (integer value) set it to the delay in milliseconds, this will the delay part of the SO_LINGER configuration.
SOCKET_DEFAULT_QUICKACK boolean
SOCKET_DEFAULT_NODELAY boolean

Also, some options exist to specify uncommon installation paths. Their format is ${DEP}_INCDIR or ${DEP}_LIBDIR, and DEP might take the given values ASN1C, ATTR, CURL, JSONC, LEVELDB, ZK, ZLIB, ZMQ

We recommend that you specify the installation directory (especially if you are not root) at this step so you don't need to repeat it when calling make install:

cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local [OTHER CMAKE PARAMETERS] ${SRCDIR}

Building

Now that cmake succeeded, it is time to build and install the binaries with make.

make
make test
make install  # or make DESTDIR=${install_dir} install

We suggest to install Python dependencies in a virtualenv instead of directly on the system.

python3 -m venv oiovenv
# or "virtualenv -p /usr/bin/python3 oiovenv"
source oiovenv/bin/activate

Then install the python module inside your virtualenv:

pip install -e ${SRCDIR}
${SRCDIR}/tools/patch-python-modules.sh

Then install FoundationDB with oio-install-fdb.sh

./tools/oio-install-fdb.sh

A lot of variables are available, consider reading Variables.md for more information.

Step by step

FoundationDB install

  • install et conf FondationDB (cf script oio-sds/tools/oio-install-fdb.sh)

Zookeeper install

  • sudo apt-get install libzookeeper-mt-dev zookeeper

Golang install

  • install golang version 1.22.5 and place it where you want (I did in ~/bin)

Ubuntu packages install

  • sudo apt-get install -y $(tr '\n' ' ' < .cds/deps-ubuntu-focal.txt)

Prepare venv to install packages inside

  • python3 -m venv venv (python3 version 3.10.12)
  • source venv/bin/activate

Build repo

  • mkdir build && cd build
  • mkdir ~/local
  • export SDS="~/local"
  • export PATH="$PATH:/~/bin/go/bin/:~/local/bin" < change go/bin directory if not in ~/bin
  • cmake -DCMAKE_INSTALL_PREFIX=${SDS} -DLD_LIBDIR=lib -DCMAKE_BUILD_TYPE=Debug ..
  • make install -j
  • cd ..

Start redpanda

  • export OIO_NS=OPENIO
  • export OIO_ACCOUNT=AUTH_demo
  • start redpanda from oio-sds/docker -> docker-compose -f redpanda.docker-compose.yml up -d

Install python3 packages

  • pip install .
  • pip install -r test-requirements.txt

Reset stack and start it all

  • oio-reset.sh -f oio-sds/etc/bootstrap-preset-SINGLE.yml -r RegionOne -U

Check if openio responds correctly

  • openio cluster show

Check the services in systemctl

  • systemctl --user list-dependencies oio-cluster.target
    (alternative would be openioctl.sh -c status|status2)