Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update build workflow to build indexer plugins #360

Merged
merged 28 commits into from
Aug 28, 2024
Merged
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
3446503
Update build workflow to include Wazuh plugins
AlexRuiz7 Aug 16, 2024
a06bed7
Try new approach to build wazuh-indexer with plugins
AlexRuiz7 Aug 16, 2024
20eddf0
Remove old code
AlexRuiz7 Aug 16, 2024
167809a
Remove ADMINS.md
AlexRuiz7 Aug 16, 2024
6ebc18f
Sync mavel local path across jobs
AlexRuiz7 Aug 20, 2024
1eb2403
Fix versioning of wazuh-indexer-plugins
AlexRuiz7 Aug 20, 2024
c693b68
Fix versioning of wazuh-indexer-plugins
AlexRuiz7 Aug 20, 2024
c444752
Pass version and revision to publishToMavenLocal
AlexRuiz7 Aug 20, 2024
7b74df4
Add version check test
AlexRuiz7 Aug 21, 2024
b4a61fa
Format files
AlexRuiz7 Aug 21, 2024
979407e
Use upload-artifact and download-artifact to share the plugins' zips …
AlexRuiz7 Aug 22, 2024
12eecba
Add repo path
AlexRuiz7 Aug 22, 2024
39d123f
Fix plugin name
AlexRuiz7 Aug 22, 2024
38ecf65
Roll back
AlexRuiz7 Aug 22, 2024
d4219f8
Remove exit 1
AlexRuiz7 Aug 22, 2024
4e5d0ea
Fix relative path to the plugins
AlexRuiz7 Aug 22, 2024
e415653
List plugins folder
AlexRuiz7 Aug 22, 2024
dd0560e
Fix relative path
AlexRuiz7 Aug 22, 2024
3feb43f
again
AlexRuiz7 Aug 22, 2024
758e572
Change relative path to absolute
AlexRuiz7 Aug 23, 2024
b9fa45b
Clean code
AlexRuiz7 Aug 23, 2024
7a3f180
Update README.ms
AlexRuiz7 Aug 23, 2024
667af71
Apply naming convention
AlexRuiz7 Aug 27, 2024
f74dd34
Add breif steps to build wazuh-indexer with plugins
AlexRuiz7 Aug 27, 2024
e9c3389
Skip job to build plugins on no input
AlexRuiz7 Aug 28, 2024
a031b89
Improve conditional
AlexRuiz7 Aug 28, 2024
ed86236
Remove build-plugins job from build's job dependencies
AlexRuiz7 Aug 28, 2024
d4c8bf6
Roll back
AlexRuiz7 Aug 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update README.ms
AlexRuiz7 committed Aug 23, 2024

Verified

This commit was signed with the committer’s verified signature.
AlexRuiz7 Álex Ruiz
commit 7a3f1807a1db47de2c85089b5f4684b52d46d039
175 changes: 87 additions & 88 deletions packaging_scripts/README.md
Original file line number Diff line number Diff line change
@@ -22,7 +22,7 @@ The names of the packages are managed by the `baptizer.sh` script.

## Build and Assemble in Act

Use Act to run the `build.yml` workflow locally. The `act.input.env` file contains the inputs
Use Act to run the `build.yml` workflow locally. The `act.input.env` file contains the inputs
for the workflow. As the workflow clones the `wazuh-indexer-plugins` repository, the `GITHUB_TOKEN`
is required. You can use the `gh` CLI to authenticate, as seen in the example below.

@@ -32,7 +32,7 @@ act -j build -W .github/workflows/build.yml --artifact-server-path ./artifacts -

## Build

For local package generation, use the `build.sh` script. Take a look at the `build.yml`
For local package generation, use the `build.sh` script. Take a look at the `build.yml`
workflow file for an example of usage.

```bash
@@ -52,7 +52,7 @@ act -j build -W .github/workflows/build.yml --artifact-server-path ./artifacts
Using the [Docker environment](../docker):

```console
docker exec -it wi-build_$(<VERSION) bash packaging_scripts/build.sh -a {x64|arm64} -d {rpm|deb|tar}
docker exec -it wi-build_$(<VERSION) bash packaging_scripts/build.sh -a x64 -d tar -n $(bash packaging_scripts/baptizer.sh -a x64 -d tar -m)
```

The generated package is sent to the `wazuh-indexer/artifacts` folder.
@@ -62,7 +62,6 @@ The generated package is sent to the `wazuh-indexer/artifacts` folder.
**Note:** set the environment variable `TEST=true` to assemble a package with the required plugins only,
speeding up the assembly process.


### TAR

The assembly process for tarballs consists on:
@@ -82,66 +81,66 @@ For DEB packages, the `assemble.sh` script will perform the following operations

1. Extract the deb package using `ar` and `tar` tools.

> By default, `ar` and `tar` tools expect the package to be in `wazuh-indexer/artifacts/tmp/deb`.
> The script takes care of creating the required folder structure, copying also the min package
> and the Makefile.

Current folder loadout at this stage:

```
artifacts/
|-- dist
| |-- wazuh-indexer-min_4.9.1_amd64.deb
`-- tmp
`-- deb
|-- Makefile
|-- data.tar.gz
|-- debmake_install.sh
|-- etc
|-- usr
|-- var
`-- wazuh-indexer-min_4.9.1_amd64.deb
```

`usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.deb`.
`Makefile` and the `debmake_install` are copied over from `wazuh-indexer/distribution/packages/src/deb`.
The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file.
> By default, `ar` and `tar` tools expect the package to be in `wazuh-indexer/artifacts/tmp/deb`.
> The script takes care of creating the required folder structure, copying also the min package
> and the Makefile.

Current folder loadout at this stage:

```
artifacts/
|-- dist
| |-- wazuh-indexer-min_4.9.1_amd64.deb
`-- tmp
`-- deb
|-- Makefile
|-- data.tar.gz
|-- debmake_install.sh
|-- etc
|-- usr
|-- var
`-- wazuh-indexer-min_4.9.1_amd64.deb
```

`usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.deb`.
`Makefile` and the `debmake_install` are copied over from `wazuh-indexer/distribution/packages/src/deb`.
The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file.

2. Install the plugins using the `opensearch-plugin` CLI tool.
3. Set up configuration files.

> Included in `min-package`. Default files are overwritten.
> Included in `min-package`. Default files are overwritten.

4. Bundle a DEB file with `debmake` and the `Makefile`.

> `debmake` and other dependencies can be installed using the `provision.sh` script.
> The script is invoked by the GitHub Workflow.

Current folder loadout at this stage:

```
artifacts/
|-- artifact_name.txt
|-- dist
| |-- wazuh-indexer-min_4.9.1_amd64.deb
| `-- wazuh-indexer_4.9.1_amd64.deb
`-- tmp
`-- deb
|-- Makefile
|-- data.tar.gz
|-- debmake_install.sh
|-- etc
|-- usr
|-- var
|-- wazuh-indexer-min_4.9.1_amd64.deb
`-- debian/
| -- control
| -- copyright
| -- rules
| -- preinst
| -- prerm
| -- postinst
```
> `debmake` and other dependencies can be installed using the `provision.sh` script.
> The script is invoked by the GitHub Workflow.

Current folder loadout at this stage:

```
artifacts/
|-- artifact_name.txt
|-- dist
| |-- wazuh-indexer-min_4.9.1_amd64.deb
| `-- wazuh-indexer_4.9.1_amd64.deb
`-- tmp
`-- deb
|-- Makefile
|-- data.tar.gz
|-- debmake_install.sh
|-- etc
|-- usr
|-- var
|-- wazuh-indexer-min_4.9.1_amd64.deb
`-- debian/
| -- control
| -- copyright
| -- rules
| -- preinst
| -- prerm
| -- postinst
```

#### Running in Docker

@@ -152,7 +151,7 @@ Pre-requisites:
- Using the [Docker environment](../docker):

```console
docker exec -it wi-assemble_$(<VERSION) bash packaging_scripts/assemble.sh -a x64 -d deb
docker exec -it wi-assemble_$(<VERSION) bash packaging_scripts/assemble.sh -a x64 -d deb -r 1
```

### RPM
@@ -165,45 +164,45 @@ The script will:

1. Extract the RPM package using `rpm2cpio` and `cpio` tools.

> By default, `rpm2cpio` and `cpio` tools expect the package to be in `wazuh-indexer/artifacts/tmp/rpm`.The script takes care of creating the required folder structure, copying also the min package and the SPEC file.
> By default, `rpm2cpio` and `cpio` tools expect the package to be in `wazuh-indexer/artifacts/tmp/rpm`. The script takes care of creating the required folder structure, copying also the min package and the SPEC file.

Current folder loadout at this stage:
Current folder loadout at this stage:

```
/rpm/$ARCH
/etc
/usr
/var
wazuh-indexer-min-*.rpm
wazuh-indexer.rpm.spec
```
```
/rpm/$ARCH
/etc
/usr
/var
wazuh-indexer-min-*.rpm
wazuh-indexer.rpm.spec
```

`usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.rpm`.
`wazuh-indexer.rpm.spec` is copied over from `wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec`.
The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file.
`usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.rpm`.
`wazuh-indexer.rpm.spec` is copied over from `wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec`.
The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file.

2. Install the plugins using the `opensearch-plugin` CLI tool.
3. Set up configuration files.

> Included in `min-package`. Default files are overwritten.
> Included in `min-package`. Default files are overwritten.

4. Bundle an RPM file with `rpmbuild` and the SPEC file `wazuh-indexer.rpm.spec`.

> `rpmbuild` is part of the `rpm` OS package.
> `rpmbuild` is part of the `rpm` OS package.

> `rpmbuild` is invoked from `wazuh-indexer/artifacts/tmp/rpm`. It creates the {BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} folders and applies the rules in the SPEC file. If successful, `rpmbuild` will generate the package in the `RPMS/` folder. The script will copy it to `wazuh-indexer/artifacts/dist` and clean: remove the `tmp\` folder and its contents.
> `rpmbuild` is invoked from `wazuh-indexer/artifacts/tmp/rpm`. It creates the {BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} folders and applies the rules in the SPEC file. If successful, `rpmbuild` will generate the package in the `RPMS/` folder. The script will copy it to `wazuh-indexer/artifacts/dist` and clean: remove the `tmp\` folder and its contents.

Current folder loadout at this stage:
Current folder loadout at this stage:

```
/rpm/$ARCH
/{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}
/etc
/usr
/var
wazuh-indexer-min-*.rpm
wazuh-indexer.rpm.spec
```
```
/rpm/$ARCH
/{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}
/etc
/usr
/var
wazuh-indexer-min-*.rpm
wazuh-indexer.rpm.spec
```

#### Running in Docker

@@ -214,7 +213,7 @@ Pre-requisites:
- Using the [Docker environment](../docker):

```console
docker exec -it wi-assemble_$(<VERSION) bash packaging_scripts/assemble.sh -a x64 -d rpm
docker exec -it wi-assemble_$(<VERSION) bash packaging_scripts/assemble.sh -a x64 -d rpm -r 1
```

## Bash scripts reference
@@ -234,7 +233,7 @@ scripts:
name: the name of the package to be generated.
outputs:
package: minimal wazuh-indexer package for the required distribution.

- file: assemble.sh
description: |
bundles the wazuh-indexer package generated in by build.sh with plugins,
@@ -245,10 +244,10 @@ scripts:
revision: revision number. 0 by default.
outputs:
package: wazuh-indexer package.

- file: provision.sh
description: Provision script for the assembly of DEB packages.

- file: baptizer.sh
description: generate the wazuh-indexer package name depending on the parameters.
inputs: