Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge 4.10.2 into master #514

Merged
merged 100 commits into from
Nov 5, 2024
Merged
Changes from 1 commit
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
9ad2f82
Init wazuh-indexer (#3)
AlexRuiz7 Aug 3, 2023
352e675
Create codeql.yml
AlexRuiz7 Aug 3, 2023
91fef34
Update dependabot.yml
AlexRuiz7 Aug 3, 2023
ce79f5a
Update SECURITY.md (#30)
AlexRuiz7 Sep 19, 2023
bf09e9b
Add ECS mappings generator (#36)
AlexRuiz7 Oct 9, 2023
a614448
Add default query fields to vulnerability detector index (#40)
AlexRuiz7 Oct 20, 2023
6722947
Create gradle_build.yml
AlexRuiz7 Oct 20, 2023
d376486
Update gradle_build.yml
AlexRuiz7 Oct 20, 2023
91a2de0
Add a script to configure the rollover policy (#49)
AlexRuiz7 Nov 2, 2023
0369a4e
Update ISM init script (#50)
AlexRuiz7 Nov 7, 2023
2e7f6d4
Fix bug with -i option (#51)
AlexRuiz7 Nov 8, 2023
9d5e91c
Update min_doc_count value (#52)
AlexRuiz7 Nov 14, 2023
a5f309d
Improve ISM init script (#57)
AlexRuiz7 Nov 15, 2023
bf4d828
Update distribution files (#59)
AlexRuiz7 Nov 22, 2023
b07b964
Update documentation of the ECS tooling (#67)
AlexRuiz7 Nov 28, 2023
5607ca0
Add workflow for package generation (#65)
AlexRuiz7 Nov 28, 2023
df8760e
Add docker compose environment (#66)
AlexRuiz7 Nov 29, 2023
aef0064
Rename packages to wazuh-indexer (#69)
AlexRuiz7 Dec 5, 2023
692ee6a
Update vulnerability index mappings (#75)
AlexRuiz7 Dec 7, 2023
f6c9a3c
Update `indexer-ism-init.sh` (#81)
AlexRuiz7 Dec 18, 2023
693c074
Add workflow to assemble packages (#85)
AlexRuiz7 Dec 26, 2023
458c7ee
Fix yellow cluster state (#95)
AlexRuiz7 Dec 27, 2023
62d4295
Update ism-init script (#97)
AlexRuiz7 Jan 3, 2024
3b126b8
Add tools to assemble DEB packages (#96)
AlexRuiz7 Jan 4, 2024
483f4c5
Update README.md
AlexRuiz7 Jan 4, 2024
fba5a68
Build scripts and GH workflows artifacts naming fix (#112)
f-galland Jan 10, 2024
2dfe8e9
Use short SHA as Git reference in packages naming (#100)
f-galland Jan 10, 2024
c85f426
Remove unneeded files from assembled packages (#115)
f-galland Jan 12, 2024
347103e
Add missing tools and files back into Wazuh Indexer packages (#117)
f-galland Jan 12, 2024
25c9179
Remove unneeded symbolic links from assembled packages (#121)
f-galland Jan 15, 2024
d10c450
Update issue templates (#127)
AlexRuiz7 Jan 16, 2024
fca8376
Fix RPM package references to /var/run (#119)
f-galland Jan 18, 2024
bc9546c
Removing post-install message from wazuh-indexer.rpm.spec (#131)
f-galland Jan 18, 2024
828c2f8
Add tests to the packages building process (#132)
AlexRuiz7 Jan 18, 2024
1ba2351
Get Wazuh version from VERSION file (#122)
f-galland Jan 19, 2024
3fe6905
Removing /usr/share/lintian/overrides/wazuh-indexer from deb packages…
f-galland Jan 19, 2024
b9b0aa0
Add `wazuh-template.json` to packages (#116)
f-galland Jan 19, 2024
724b7a5
Adding Debian packaging config files from Opensearch (#118)
f-galland Jan 19, 2024
c142fcd
Fix Build workflow to run on push events (#134)
AlexRuiz7 Jan 19, 2024
e89f567
Use maven for plugin download (#139)
f-galland Jan 22, 2024
23bf3a1
Add new custom field to the vulnerability detector index (#141)
AlexRuiz7 Jan 25, 2024
fb01cc0
Fine tuning permissions on assembled packages (#137)
f-galland Jan 30, 2024
3e7c582
Init. Amazon Security Lake integration (#143)
AlexRuiz7 Jan 31, 2024
c3a9d49
Add events generator tool for `wazuh-alerts` (#152)
AlexRuiz7 Feb 15, 2024
9024768
Add `wazuh.manager.name` to VD mappings (#158)
AlexRuiz7 Feb 20, 2024
4d9f2a5
Create compatibility_request.md (#163)
AlexRuiz7 Feb 23, 2024
de40567
Add Python module to accomplish OCSF compliant events (#159)
AlexRuiz7 Mar 4, 2024
092874c
Update Gradle setup action (#182)
AlexRuiz7 Mar 7, 2024
8e4d75b
Update vulnerability-states fields (#177)
AlexRuiz7 Mar 7, 2024
f152f81
Automate package's testing (#178)
AlexRuiz7 Mar 8, 2024
30f7084
Remove ecs.version from query.default_fields (#184)
AlexRuiz7 Mar 8, 2024
9eeb248
Upload packages to S3 (#179)
AlexRuiz7 Mar 8, 2024
8f07f88
Add bash to Docker dev image (#185)
AlexRuiz7 Mar 15, 2024
fa72a21
Update wazuh-states-vulnerabilities index mapping (#191)
AlexRuiz7 Mar 26, 2024
b6c98d6
Add pipeline to generate release packages (#193)
AlexRuiz7 Mar 27, 2024
43cc0d7
Build Docker images (#194)
AlexRuiz7 Apr 4, 2024
9af6fe8
Add on.workflow_call to build_single.yml workflow (#200)
AlexRuiz7 Apr 9, 2024
b936fe6
Add Pyhton module to implement Amazon Security Lake integration (#186)
AlexRuiz7 Apr 9, 2024
224a291
Replace choice with string on workflow_call (#207)
AlexRuiz7 Apr 18, 2024
0ca9f27
Use AWS_REGION secret (#209)
AlexRuiz7 Apr 24, 2024
a40b93b
Add Lambda function for the Amazon Security Lake integration (#189)
AlexRuiz7 Apr 24, 2024
bf2f55c
Bump Java version in Docker environments (#210)
AlexRuiz7 Apr 26, 2024
7fe7096
Fix access denied error during log rotation (#212)
AlexRuiz7 Apr 26, 2024
9c65d2b
Save intermediate OCSF files to an S3 bucket (#218)
AlexRuiz7 Apr 26, 2024
6127124
Fix Parquet files format (#217)
AlexRuiz7 Apr 26, 2024
d85d99f
Fix mapping to Detection Finding OCSF class (#220)
AlexRuiz7 Apr 29, 2024
d7786a3
Map events to OCSF's Security Finding class (#221)
AlexRuiz7 Apr 30, 2024
701190c
Add ID input to workflows (#229)
rauldpm May 17, 2024
31ac9a6
Add OPENSEARCH_TMPDIR variable to service and create directory in pac…
f-galland May 21, 2024
6de22a1
Improve workflow's run-name with tagret system and architeture (#237)
AlexRuiz7 May 28, 2024
4efe0a2
Add documentation for the Amazon Security Lake integration (#226)
AlexRuiz7 May 28, 2024
909a9e2
Rename environment variable (#240)
AlexRuiz7 May 28, 2024
816fa2d
Remove maintainer-approval.yml (#241)
AlexRuiz7 May 28, 2024
f5d00c2
Improve logging and error handling on ASL Lambda function (#242)
AlexRuiz7 May 29, 2024
ec9fd89
Update .gitattributes (#243)
AlexRuiz7 May 29, 2024
7a665ae
Change . for : in debian's postinst (#245)
f-galland May 31, 2024
12311e8
Add integration with Elastic (#248)
AlexRuiz7 Jun 5, 2024
c5d13aa
Added S3 URI output to package generation upload (#249)
rauldpm Jun 7, 2024
e1d0334
Add OpenSearch integration (#258)
f-galland Jun 7, 2024
5e4c84f
Add Splunk integration (#257)
AlexRuiz7 Jun 11, 2024
aae3b6f
Add Manager to Elastic integration (#266)
AlexRuiz7 Jun 12, 2024
f4cc1e6
Add Manager to Splunk integration (#268)
AlexRuiz7 Jun 14, 2024
5a7445a
Add Manager to OpenSearch integration (#267)
AlexRuiz7 Jun 14, 2024
4609871
Attempt nr.2 to fix #277 (#280)
AlexRuiz7 Jun 25, 2024
b32fa76
Remove references to indexer-ism-init.sh and wazuh-template.json (#281)
f-galland Jun 25, 2024
95f1e12
Bump 4.10.0 (#272)
AlexRuiz7 Jun 20, 2024
b4d103d
Merge 4.9.1 into 4.10.0 (#358)
AlexRuiz7 Aug 20, 2024
b8c8bab
Merge 4.9.2 into 4.10.0 (#378)
AlexRuiz7 Sep 6, 2024
d616b80
Fix build.gradle (#381)
AlexRuiz7 Sep 9, 2024
f53e992
Remove old compose files for integrations (#386)
AlexRuiz7 Sep 9, 2024
095d2e3
Delete integrations/docker/amazon-security-lake.yml
AlexRuiz7 Sep 9, 2024
c629334
Delete integrations/docker/config directory
AlexRuiz7 Sep 9, 2024
066c12a
Update vulnerability detector index template (#383)
AlexRuiz7 Sep 9, 2024
ce1c6b2
Merge 4.9.1 into 4.10.0 (#426)
AlexRuiz7 Sep 23, 2024
aad4754
Bump version to 4.10.1 (#430)
AlexRuiz7 Sep 24, 2024
ed170ac
Support new version 4.10.2 (#441)
AlexRuiz7 Oct 3, 2024
abe5f5f
Enable assembly of ARM packages (#444)
AlexRuiz7 Oct 4, 2024
e6e60cd
Merge 4.10.1 into 4.10.2 (#473)
AlexRuiz7 Oct 17, 2024
a7bbb60
Merge 4.10.1 into 4.10.2 (#513)
AlexRuiz7 Nov 5, 2024
f9d9a2b
Merge branch '4.10.2' into merge-4.10.2-into-master
AlexRuiz7 Nov 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add pipeline to generate release packages (#193)
* Add script to get the version of OpenSearch

* Set revision to 0 by default.

- Reduce inputs for scripts.
- Add script to generate packages' naming convention.
- Make scripts self-aware of the OpenSearch version.

* Fix assemble

* Smoke test new pipeline to build packages

* Fix syntax errors

* Update build.yml

Signed-off-by: Álex Ruiz <[email protected]>

* Add workflow to build packages on push

* Run actionlint

* Fix jq argjson

* Fix set matrix output ?

* Try new approach using a single workflow

* Fix GITHUB_OUTPUT

* Fix baptizer invocation

* Add testing and upload to new approach

* Fix hard coded revision number on RPM assembly

* New attempt

* Skip upload unless specified

* Install plugins on RPM

* Promote new approach

Removes previous workflows to generate packages

* Fix workflow name

* Attempt to fix release package naming

* Fix build.sh invocation from workflow

* Use min package name in workflow

* Use min package name for release naming convention in workflow

* Attemtp to fix regex

* Upgrade to aws-actions/configure-aws-credentials@v4

Clean up

* Apply latest requirements

Add workflow with single matrix for QA use. Rename inputs. Add checksum input.

* Add checksum generation and upload

* Use choice as input types for system and architecture

* Invoke build single packages with upload option

* Add documentation and clean up

* Rename scripts folder to packaging_scripts

---------

Signed-off-by: Álex Ruiz <[email protected]>
AlexRuiz7 committed Sep 9, 2024

Verified

This commit was signed with the committer’s verified signature.
AlexRuiz7 Álex Ruiz
commit b6c98d603a0b3441fd9eb7276f193dfa52eafebd
221 changes: 163 additions & 58 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -1,23 +1,68 @@
name: Build packages
name: Build packages (on demand)

# This workflow runs when any of the following occur:
# - Run manually
# - Invoked from another workflow
on:
push:
# Sequence of patterns matched against refs/heads
branches:
- "ci/*"
workflow_dispatch:
inputs:
revision:
description: "Revision"
type: string
required: true
default: "1"
default: "0"
upload:
description: "Upload ?"
type: bool
type: boolean
default: false
is_stage:
description: "Is stage ?"
type: boolean
default: false
distribution:
description: '[ "tar", "rpm", "deb" ]'
type: string
default: '[ "rpm", "deb" ]'
architecture:
description: '[ "x64", "arm64" ]'
type: string
default: '[ "x64" ]'
checksum:
description: "Checksum ?"
type: boolean
default: false
workflow_call:
inputs:
revision:
description: "Revision"
type: string
default: "0"
upload:
description: "Upload ?"
type: boolean
default: false
is_stage:
description: "Is stage ?"
type: boolean
default: false
distribution:
description: '[ "tar", "rpm", "deb" ]'
type: string
default: '[ "rpm", "deb" ]'
architecture:
description: '[ "x64", "arm64" ]'
type: string
default: '[ "x64" ]'
checksum:
description: "Checksum ?"
type: boolean
default: false
secrets:
CI_INTERNAL_DEVELOPMENT_BUCKET_USER_ACCESS_KEY:
required: true
description: "AWS user access key"
CI_INTERNAL_DEVELOPMENT_BUCKET_USER_SECRET_KEY:
required: true
description: "AWS user secret key"

# ==========================
# Bibliography
@@ -33,57 +78,117 @@ on:
# | https://docs.github.com/en/actions/learn-github-actions/expressions#example

jobs:
version:
uses: ./.github/workflows/r_version.yml

commit_sha:
uses: ./.github/workflows/r_commit_sha.yml
matrix:
name: Set up matrix
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.setup.outputs.matrix }}
steps:
- id: setup
run: |
matrix=$(jq -cn \
--argjson distribution '${{ inputs.distribution }}' \
--argjson architecture '${{ inputs.architecture }}' \
'{distribution: $distribution, architecture: $architecture}'
)
echo "matrix=$matrix" >> $GITHUB_OUTPUT
build:
needs: [version, commit_sha]
strategy:
matrix:
distribution: [tar, rpm, deb]
architecture: [x64, arm64]
uses: ./.github/workflows/r_build.yml
with:
architecture: ${{ matrix.architecture }}
distribution: ${{ matrix.distribution }}
revision: ${{ github.event_name == 'push' && '1' || inputs.revision }}
name: wazuh-indexer-min_${{ needs.version.outputs.version }}-${{ github.event_name == 'push' && '1' || inputs.revision }}-${{ matrix.architecture }}_${{ needs.commit_sha.outputs.commit_sha }}.${{ matrix.distribution }}

assemble:
needs: [version, commit_sha, build]
strategy:
matrix:
distribution: [tar, rpm, deb]
architecture: [x64, arm64]
exclude:
# skip arm64 until we have arm runners
- architecture: arm64
- distribution: tar

uses: ./.github/workflows/r_assemble.yml
with:
architecture: ${{ matrix.architecture }}
distribution: ${{ matrix.distribution }}
min: wazuh-indexer-min_${{ needs.version.outputs.version }}-${{ github.event_name == 'push' && '1' || inputs.revision }}-${{ matrix.architecture }}_${{ needs.commit_sha.outputs.commit_sha }}.${{ matrix.distribution }}

test:
needs: [version, commit_sha, assemble]
needs: [matrix]
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
os: [{ suffix: "amd64", ext: "deb" }, { suffix: "x86_64", ext: "rpm" }]
uses: ./.github/workflows/r_test.yml
with:
package: wazuh-indexer-${{ needs.version.outputs.version }}-${{ github.event_name == 'push' && '1' || inputs.revision }}_${{ matrix.os.suffix }}_${{ needs.commit_sha.outputs.commit_sha }}.${{ matrix.os.ext }}

upload:
needs: [version, commit_sha, test]
# Upload only on 'workflow_dispatch' event and if 'upload=true'
if: ${{ github.event_name == 'push' && inputs.upload }}
uses: ./.github/workflows/r_upload.yml
with:
package: wazuh-indexer-${{ needs.version.outputs.version }}-${{ github.event_name == 'push' && '1' || inputs.revision }}_${{ matrix.os.suffix }}_${{ needs.commit_sha.outputs.commit_sha }}.${{ matrix.os.ext }}
secrets: inherit
matrix: ${{ fromJson(needs.matrix.outputs.matrix) }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: 11

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v3

- name: Provision
if: ${{ matrix.distribution == 'deb' }}
run: |
sudo bash packaging_scripts/provision.sh
- name: Run `baptizer.sh` (min)
run: |
name=$(bash packaging_scripts/baptizer.sh -m \
-a ${{ matrix.architecture }} \
-d ${{ matrix.distribution }} \
-r ${{ inputs.revision }} \
${{ inputs.is_stage && '-x' || '' }} \
)
echo "name=$name" >> $GITHUB_OUTPUT
id: min_package

- name: Run `baptizer.sh`
run: |
name=$(bash packaging_scripts/baptizer.sh \
-a ${{ matrix.architecture }} \
-d ${{ matrix.distribution }} \
-r ${{ inputs.revision }} \
${{ inputs.is_stage && '-x' || '' }} \
)
echo "name=$name" >> $GITHUB_OUTPUT
id: package

- name: Run `build.sh`
run: |
bash packaging_scripts/build.sh \
-a ${{ matrix.architecture }} \
-d ${{ matrix.distribution }} \
-n ${{ steps.min_package.outputs.name }}
- name: Run `assemble.sh`
run: |
bash packaging_scripts/assemble.sh \
-a ${{ matrix.architecture }} \
-d ${{ matrix.distribution }} \
-r ${{ inputs.revision }}
- name: Test RPM package
if: ${{ matrix.distribution == 'rpm' }}
uses: addnab/docker-run-action@v3
with:
image: redhat/ubi9:latest
options: -v ${{ github.workspace }}/artifacts/dist:/artifacts/dist
run: |
yum localinstall "/artifacts/dist/${{ steps.package.outputs.name }}" -y
- name: Test DEB package
if: ${{ matrix.distribution == 'deb' }}
run: |
sudo dpkg -i "artifacts/dist/${{ steps.package.outputs.name }}"
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: ${{ steps.package.outputs.name }}
path: artifacts/dist/${{ steps.package.outputs.name }}
if-no-files-found: error

- name: Set up AWS CLI
if: ${{ inputs.upload }}
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.CI_INTERNAL_DEVELOPMENT_BUCKET_USER_ACCESS_KEY }}
aws-secret-access-key: ${{ secrets.CI_INTERNAL_DEVELOPMENT_BUCKET_USER_SECRET_KEY }}
aws-region: us-east-1

- name: Upload package to S3
if: ${{ inputs.upload }}
run: |
src="artifacts/dist/${{ steps.package.outputs.name }}"
dest="s3://packages-dev.internal.wazuh.com/development/wazuh/4.x/main/packages/"
aws s3 cp "$src" "$dest"
- name: Upload checksum to S3
if: ${{ inputs.upload && inputs.checksum }}
run: |
src="artifacts/dist/${{ steps.package.outputs.name }}.sha512"
dest="s3://packages-dev.internal.wazuh.com/development/wazuh/4.x/main/packages/"
aws s3 cp "$src" "$dest"
14 changes: 14 additions & 0 deletions .github/workflows/build_on_push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: Build packages (on push)

# This workflow runs when any of the following occur:
# - On push to branches named after ci/*
on:
push:
# Sequence of patterns matched against refs/heads
branches:
- "ci/*"

jobs:
call-build-workflow:
uses: ./.github/workflows/build.yml
secrets: inherit
46 changes: 46 additions & 0 deletions .github/workflows/build_single.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: Build packages (single)

# This workflow runs when any of the following occur:
# - Run manually
on:
workflow_dispatch:
inputs:
revision:
description: "Revision"
type: string
default: "0"
checksum:
description: "Checksum ?"
type: boolean
default: false
is_stage:
description: "Is stage ?"
type: boolean
default: false
system:
description: "Package OS"
type: choice
options:
- rpm
- deb
default: deb
architecture:
description: "Package architecture"
type: choice
options:
- amd64
- x86_64
default: amd64

jobs:
call-build-workflow:
uses: ./.github/workflows/build.yml
with:
revision: ${{ inputs.revision }}
checksum: ${{ inputs.checksum }}
is_stage: ${{ inputs.is_stage }}
distribution: '[ "${{ inputs.system }}" ]'
upload: true
# Architecture is always 'x64', which is the default value in ./build.yml
# It is an input just for convenience and standardisation.
secrets: inherit
61 changes: 0 additions & 61 deletions .github/workflows/r_assemble.yml

This file was deleted.

Loading