Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify build process using a Docker container #560

Draft
wants to merge 19 commits into
base: master
Choose a base branch
from
Draft

Simplify build process using a Docker container #560

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
4c6c712
Add simple Dockerfile for the simplified package builder
QU3B1M Nov 21, 2024
cc5ddde
Add Java and Gradle installation to build Dockerfile
QU3B1M Nov 21, 2024
5bd022a
Add indexer repositories branches as arguments and repo cloning
QU3B1M Nov 21, 2024
96d497f
Add package building stages
QU3B1M Nov 21, 2024
1872bac
Add plugins and reporting built and fix ownership
QU3B1M Nov 25, 2024
ed6ea6d
Add version definition on each RUN step
QU3B1M Nov 25, 2024
038d77d
Simplify plugins build steps
QU3B1M Nov 25, 2024
d9569d3
Split the dockerfile into two files, a Dockerfile and a shell script …
QU3B1M Nov 26, 2024
1f596d6
Update builder Dockerfile entrypoint script
QU3B1M Nov 26, 2024
971682d
Add new line at .dockerignore EOF
QU3B1M Nov 26, 2024
dc6686f
Merge branch 'master' into ci/522-unify-build-process
QU3B1M Nov 26, 2024
1d1566c
Update Docker shared volume path
QU3B1M Nov 26, 2024
1191e5e
Remove old package generation Docker image
QU3B1M Nov 26, 2024
adefd8e
Update package building documentation
QU3B1M Nov 26, 2024
5c1b7a1
Fix typos on the documentation
QU3B1M Nov 26, 2024
3f9521b
Update default path to store the package on documentation
QU3B1M Nov 26, 2024
ab2eefb
Move artifacts directory creation to previous stage
QU3B1M Nov 27, 2024
dd08f88
store changes
f-galland Nov 28, 2024
b277058
Add compose and script
f-galland Nov 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
218 changes: 111 additions & 107 deletions build-scripts/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
> This document is pending a review. Let us know if you find any issues.

The packages' generation process consists on 2 steps:

[build.yml](../.github/workflows/build.yml)
- **Build**: compiles the Java application and bundles it into a package.
- **Assembly**: uses the package from the previous step and inflates it with plugins and
configuration files, ready for production deployment.
Expand All @@ -15,64 +15,138 @@ We usually generate the packages using GitHub Actions, however, the process is d
be independent enough for maximum portability. GitHub Actions provides infrastructure, while
the building process is self-contained in the application code.

Each section includes instructions to generate packages locally, using Act or Docker.

- [Install Act](https://github.com/nektos/act)
This guide includes instructions to generate packages locally, using Act or Docker.

The names of the packages are managed by the `baptizer.sh` script.

## Building the plugins
## Local environment

### Build

1. Build a wazuh-indexer minimal package
```bash
bash build-scripts/build.sh -a [ARCHITECTURE] -d [PACKAGE_TYPE] -n $(bash build-scripts/baptizer.sh -a [ARCHITECTURE] -d [PACKAGE_TYPE] -m)
```
> A basic package including only the Wazuh Indexer engine without extra plugin is generated first using the `build.sh` script.
> Take a look at the `build.yml` workflow file for an example of usage.
2. Build the wazuh-indexer plugins. (Repeat for each plugin `command-manager`, `setup`, and `reporting`)
```bash
cd path/to/wazuh/indexer/plugin/
./gradlew build -Dversion=[VERSION] -Drevision=[REVISION]
```
> Follow the [DEVELOPER_GUIDE.md](https://github.com/wazuh/wazuh-indexer-plugins/blob/master/DEVELOPER_GUIDE.md) instructions to build the plugins. The build scripts expect the plugins in the Maven local repository or under the `artifacts/plugins` folder.
3. Publish the plugins to the local Maven repository. (Repeat for each plugin `command-manager`, `setup`, and `reporting`)
```bash
./gradlew publishToMavenLocal
```
> Alternatively, copy the generated zip files to the `artifacts/plugins` folder.
> ```bash
> cp build/distributions/wazuh-indexer-[PLUGIN_NAME]-[VERSION]-[REVISION].zip /wazuh-indexer/artifacts/plugins/
> ```

### Assemble

> **Note:** set the environment variable `TEST=true` to assemble a package with the required plugins only,
speeding up the assembly process.

Follow the [DEVELOPER_GUIDE.md](https://github.com/wazuh/wazuh-indexer-plugins/blob/master/DEVELOPER_GUIDE.md) instructions to build the plugins. The build scripts expect the plugins in the Maven local repository or under the `artifacts/plugins` folder.
1. Assemble the `wazuh-indexer` complete package
```bash
bash build-scripts/assemble.sh -a [ARCHITECTURE] -d [PACKAGE_TYPE]
```

1. Build the plugins.
2. Publish the plugins to the local Maven repository: run `./gradlew publishToMavenLocal`.
- Alternatively, copy the generated zip files to the `artifacts/plugins` folder.
3. Build and Assemble the `wazuh-indexer` package.
## Docker environment

## Build and Assemble in Act
- [Install Docker](https://docs.docker.com/engine/install/)

Use Act to run the `build.yml` workflow locally. The `act.input.env` file contains the inputs
for the workflow. As the workflow clones the `wazuh-indexer-plugins` repository, the `GITHUB_TOKEN`
is required. You can use the `gh` CLI to authenticate, as seen in the example below.
The `builder` image automates the build and assemble process for the Wazuh Indexer and its plugins, making it easy to create packages on any system.

```console
act -j build -W .github/workflows/build.yml --artifact-server-path ./artifacts --input-file build-scripts/act.input.env -s GITHUB_TOKEN="$(gh auth token)"
```
1. Build the image:
```bash
cd docker/builder && docker build -t wazuh-indexer-builder . && cd ../..
```
2. Execute the package build process
```bash
docker run --rm -v ./artifacts/dist:/home/indexer/artifacts wazuh-indexer-builder
```
The resulting package will be stored at `./artifacts/dist`.

## Build
#### Environment Variables
You can customize the build process by setting the following environment variables:

For local package generation, use the `build.sh` script. Take a look at the `build.yml`
workflow file for an example of usage.
- `INDEXER_BRANCH`: The branch to use for the Wazuh Indexer (default: `master`).
- `INDEXER_PLUGINS_BRANCH`: The branch to use for the Wazuh Indexer plugins (default: `master`).
- `INDEXER_REPORTING_BRANCH`: The branch to use for the Wazuh Indexer reporting (default: `master`).
- `REVISION`: The revision number for the build (default: `0`).
- `IS_STAGE`: Whether the build is a staging build (default: `false`).
- `DISTRIBUTION`: The distribution format for the package (default: `tar`).
- `ARCHITECTURE`: The architecture for the package (default: `x64`).

Example usage with custom environment variables:
```bash
bash build-scripts/build.sh -a x64 -d tar -n $(bash build-scripts/baptizer.sh -a x64 -d tar -m)
docker run --rm -e INDEXER_BRANCH="5.0.0" -e INDEXER_PLUGINS_BRANCH="5.0.0" -e INDEXER_REPORTING_BRANCH="5.0.0" -v ./artifacts/dist:/home/indexer/artifacts wazuh-indexer-builder
```

#### Act (GitHub Workflow locally)
## Build and Assemble in Act

```console
act -j build -W .github/workflows/build.yml --artifact-server-path ./artifacts
- [Install Act](https://github.com/nektos/act)

Use Act to run the `build.yml` workflow locally. The `act.input.env` file contains the inputs
for the workflow. As the workflow clones the `wazuh-indexer-plugins` repository, the `GITHUB_TOKEN`
is required. You can use the `gh` CLI to authenticate, as seen in the example below.

[Build slim packages/build] 🏁 Job succeeded
```bash
act -j build -W .github/workflows/build.yml --artifact-server-path ./artifacts --input-file build-scripts/act.input.env -s GITHUB_TOKEN="$(gh auth token)"
```

#### Running in Docker
## Bash scripts reference

Using the [Docker environment](../docker):
The packages' generation process is guided through bash scripts. This section list and describes
them, as well as their inputs and outputs.

```console
docker exec -it wi-build_$(<VERSION) bash build-scripts/build.sh -a x64 -d tar -n $(bash build-scripts/baptizer.sh -a x64 -d tar -m)
```
```yml
scripts:
- file: build.sh
description: |
generates a distribution package by running the appropiate Gradle task
depending on the parameters.
inputs:
architecture: [x64, arm64] # Note: we only build x86_64 packages
distribution: [tar, deb, rpm]
name: the name of the package to be generated.
outputs:
package: minimal wazuh-indexer package for the required distribution.

The generated package is sent to the `wazuh-indexer/artifacts` folder.
- file: assemble.sh
description: |
bundles the wazuh-indexer package generated in by build.sh with plugins,
configuration files and demo certificates (certificates yet to come).
inputs:
architecture: [x64, arm64] # Note: we only build x86_64 packages
distribution: [tar, deb, rpm]
revision: revision number. 0 by default.
outputs:
package: wazuh-indexer package.

## Assemble
- file: provision.sh
description: Provision script for the assembly of DEB packages.

**Note:** set the environment variable `TEST=true` to assemble a package with the required plugins only,
speeding up the assembly process.
- file: baptizer.sh
description: generate the wazuh-indexer package name depending on the parameters.
inputs:
architecture: [x64, arm64] # Note: we only build x86_64 packages
distribution: [tar, deb, rpm]
revision: revision number. 0 by default.
plugins_hash: Commit hash of the `wazuh-indexer-plugins` repository.
reporting_hash: Commit hash of the `wazuh-indexer-reporting` repository.
is_release: if set, uses release naming convention.
is_min: if set, the package name will start by `wazuh-indexer-min`. Used on the build stage.
outputs:
package: the name of the wazuh-indexer package.
```

### Assemble flow reference

### TAR
#### TAR

The assembly process for tarballs consists on:

Expand All @@ -81,11 +155,11 @@ The assembly process for tarballs consists on:
3. Add Wazuh's configuration files and tools.
4. Compress.

```console
```bash
bash build-scripts/assemble.sh -a x64 -d tar -r 1
```

### DEB
#### DEB

For DEB packages, the `assemble.sh` script will perform the following operations:

Expand Down Expand Up @@ -152,18 +226,6 @@ For DEB packages, the `assemble.sh` script will perform the following operations
| -- postinst
```

#### Running in Docker

Pre-requisites:

- Current directory: `wazuh-indexer/`
- Existing deb package in `wazuh-indexer/artifacts/dist/deb`, as a result of the _Build_ stage.
- Using the [Docker environment](../docker):

```console
docker exec -it wi-assemble_$(<VERSION) bash build-scripts/assemble.sh -a x64 -d deb -r 1
```

### RPM

The `assemble.sh` script will use the output from the `build.sh` script and use it as a
Expand Down Expand Up @@ -213,61 +275,3 @@ The script will:
wazuh-indexer-min-*.rpm
wazuh-indexer.rpm.spec
```

#### Running in Docker

Pre-requisites:

- Current directory: `wazuh-indexer/`
- Existing rpm package in `wazuh-indexer/artifacts/dist/rpm`, as a result of the _Build_ stage.
- Using the [Docker environment](../docker):

```console
docker exec -it wi-assemble_$(<VERSION) bash build-scripts/assemble.sh -a x64 -d rpm -r 1
```

## Bash scripts reference

The packages' generation process is guided through bash scripts. This section list and describes
them, as well as their inputs and outputs.

```yml
scripts:
- file: build.sh
description: |
generates a distribution package by running the appropiate Gradle task
depending on the parameters.
inputs:
architecture: [x64, arm64] # Note: we only build x86_64 packages
distribution: [tar, deb, rpm]
name: the name of the package to be generated.
outputs:
package: minimal wazuh-indexer package for the required distribution.

- file: assemble.sh
description: |
bundles the wazuh-indexer package generated in by build.sh with plugins,
configuration files and demo certificates (certificates yet to come).
inputs:
architecture: [x64, arm64] # Note: we only build x86_64 packages
distribution: [tar, deb, rpm]
revision: revision number. 0 by default.
outputs:
package: wazuh-indexer package.

- file: provision.sh
description: Provision script for the assembly of DEB packages.

- file: baptizer.sh
description: generate the wazuh-indexer package name depending on the parameters.
inputs:
architecture: [x64, arm64] # Note: we only build x86_64 packages
distribution: [tar, deb, rpm]
revision: revision number. 0 by default.
plugins_hash: Commit hash of the `wazuh-indexer-plugins` repository.
reporting_hash: Commit hash of the `wazuh-indexer-reporting` repository.
is_release: if set, uses release naming convention.
is_min: if set, the package name will start by `wazuh-indexer-min`. Used on the build stage.
outputs:
package: the name of the wazuh-indexer package.
```
35 changes: 29 additions & 6 deletions docker/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ Multipurpose Docker environments to run, test and build `wazuh-indexer`.

1. Install [Docker][docker] as per its instructions.

1. Your workstation must meet the minimum hardware requirements:
2. Your workstation must meet the minimum hardware requirements:

- 8 GB of RAM (minimum)
- 4 cores

The more resources the better ☺

1. Clone the [wazuh-indexer][wi-repo].
3. Clone the [wazuh-indexer][wi-repo].

## Development environments

Expand All @@ -29,10 +29,33 @@ Once the `wi-dev:x.y.z` container is up, attach a shell to it and run `./gradlew

## Containers to generate packages

Use the `ci/ci.sh` script to start provisioned containers to generate packages.

The `builder` image automates the build and assemble process for the Wazuh Indexer and its plugins, making it easy to create packages on any system.

### Usage
1. Build the image:
```bash
cd docker/builder && docker build -t wazuh-indexer-builder .
```
2. Execute the package build process
```bash
docker run --rm -v /path/to/local/artifacts:/home/indexer/artifacts wazuh-indexer-builder
```
> Replace `/path/to/local/artifacts` with the actual path on your host system where you want to store the resulting package.

#### Environment Variables
You can customize the build process by setting the following environment variables:

- `INDEXER_BRANCH`: The branch to use for the Wazuh Indexer (default: `master`).
- `INDEXER_PLUGINS_BRANCH`: The branch to use for the Wazuh Indexer plugins (default: `master`).
- `INDEXER_REPORTING_BRANCH`: The branch to use for the Wazuh Indexer reporting (default: `master`).
- `REVISION`: The revision number for the build (default: `0`).
- `IS_STAGE`: Whether the build is a staging build (default: `false`).
- `DISTRIBUTION`: The distribution format for the package (default: `tar`).
- `ARCHITECTURE`: The architecture for the package (default: `x64`).

Example usage with custom environment variables:
```bash
Usage: ./ci.sh {up|down|stop}
docker run --rm -e INDEXER_BRANCH="5.0.0" -e INDEXER_PLUGINS_BRANCH="5.0.0" -e INDEXER_REPORTING_BRANCH="5.0.0" -v ./artifacts/dist:/home/indexer/artifacts wazuh-indexer-builder
```

Refer to [build-scripts/README.md](../build-scripts/README.md) for details about how to build packages.
Expand All @@ -51,5 +74,5 @@ docker build --build-arg="VERSION=5.0.0" --build-arg="INDEXER_TAR_NAME=wazuh-ind
Then, start a container with:

```console
docker run -it --rm wazuh-indexer:5.0.0
docker run -it --rm wazuh-indexer:5.0.0
```
2 changes: 1 addition & 1 deletion docker/ci/images/.dockerignore → docker/builder/.dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,4 +65,4 @@ testfixtures_shared/
.ci/jobs/

# build files generated
doc-tools/missing-doclet/bin/
doc-tools/missing-doclet/bin/
Loading
Loading