Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ci #1218

Merged
merged 6 commits into from
May 6, 2024
Merged

Ci #1218

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ffcc4e0
Fix version indentation
eb4x Apr 30, 2024
85cfabe
Make molecule container names unique-ish
eb4x Apr 30, 2024
7a17bb5
Just append -db to the ancillary container
eb4x Mar 9, 2024
fad5f2e
Bump molecule and docker plugin
eb4x May 3, 2024
b905546
Display why collection install is so slow
eb4x May 3, 2024
3cef724
Pin dependencies to speed up CI
eb4x May 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions .github/workflows/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ jobs:
- debian11
- debian10
version:
- v64
- v62
- v60
- v64
- v62
- v60
scenario_name:
- default
- autopsk
Expand Down Expand Up @@ -60,11 +60,19 @@ jobs:

- name: Build the collection
run: |
# Pin versions to speed up CI
sed -i 's/ansible\.windows:\s*"\*"/ansible.windows: "2.3.0"/' galaxy.yml
sed -i 's/ansible\.netcommon:\s*"\*"/ansible.netcommon: "6.1.1"/' galaxy.yml
sed -i 's/ansible\.posix:\s*"\*"/ansible.posix: "1.5.4"/' galaxy.yml
sed -i 's/community\.general:\s*"\*"/community.general: "8.6.0"/' galaxy.yml
sed -i 's/community\.mysql:\s*"\*"/community.mysql: "3.9.0"/' galaxy.yml
sed -i 's/community\.postgresql:\s*"\*"/community.postgresql: "3.4.0"/' galaxy.yml

collection_file=$( basename $(ansible-galaxy collection build -f | awk -F" " '{print $NF}'))
echo "COLLECTION_FILE=$collection_file" >> $GITHUB_ENV

- name: Install the collection
run: ansible-galaxy collection install $COLLECTION_FILE
run: ansible-galaxy collection install -vvvv $COLLECTION_FILE

- name: Run role tests
working-directory: molecule/zabbix_agent_tests
Expand All @@ -73,4 +81,4 @@ jobs:
MY_MOLECULE_IMAGE=${{ matrix.container }}
MY_MOLECULE_VERSION=${{ matrix.version }}
MY_MOLECULE_DOCKER_COMMAND=${{ matrix.command }}
molecule -c common/molecule.yml test -s ${{ matrix.scenario_name }}
molecule -c common/molecule.yml test -s ${{ matrix.scenario_name }}
10 changes: 9 additions & 1 deletion .github/workflows/javagateway.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,19 @@ jobs:

- name: Build the collection
run: |
# Pin versions to speed up CI
sed -i 's/ansible\.windows:\s*"\*"/ansible.windows: "2.3.0"/' galaxy.yml
sed -i 's/ansible\.netcommon:\s*"\*"/ansible.netcommon: "6.1.1"/' galaxy.yml
sed -i 's/ansible\.posix:\s*"\*"/ansible.posix: "1.5.4"/' galaxy.yml
sed -i 's/community\.general:\s*"\*"/community.general: "8.6.0"/' galaxy.yml
sed -i 's/community\.mysql:\s*"\*"/community.mysql: "3.9.0"/' galaxy.yml
sed -i 's/community\.postgresql:\s*"\*"/community.postgresql: "3.4.0"/' galaxy.yml

collection_file=$( basename $(ansible-galaxy collection build -f | awk -F" " '{print $NF}'))
echo "COLLECTION_FILE=$collection_file" >> $GITHUB_ENV

- name: Install the collection
run: ansible-galaxy collection install $COLLECTION_FILE
run: ansible-galaxy collection install -vvvv $COLLECTION_FILE

- name: Run server role tests
run: >-
Expand Down
10 changes: 9 additions & 1 deletion .github/workflows/proxy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,19 @@ jobs:

- name: Build the collection
run: |
# Pin versions to speed up CI
sed -i 's/ansible\.windows:\s*"\*"/ansible.windows: "2.3.0"/' galaxy.yml
sed -i 's/ansible\.netcommon:\s*"\*"/ansible.netcommon: "6.1.1"/' galaxy.yml
sed -i 's/ansible\.posix:\s*"\*"/ansible.posix: "1.5.4"/' galaxy.yml
sed -i 's/community\.general:\s*"\*"/community.general: "8.6.0"/' galaxy.yml
sed -i 's/community\.mysql:\s*"\*"/community.mysql: "3.9.0"/' galaxy.yml
sed -i 's/community\.postgresql:\s*"\*"/community.postgresql: "3.4.0"/' galaxy.yml

collection_file=$( basename $(ansible-galaxy collection build -f | awk -F" " '{print $NF}'))
echo "COLLECTION_FILE=$collection_file" >> $GITHUB_ENV

- name: Install the collection
run: ansible-galaxy collection install $COLLECTION_FILE
run: ansible-galaxy collection install -vvvv $COLLECTION_FILE

- name: Run role tests
run: >-
Expand Down
10 changes: 9 additions & 1 deletion .github/workflows/server.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,19 @@ jobs:

- name: Build the collection
run: |
# Pin versions to speed up CI
sed -i 's/ansible\.windows:\s*"\*"/ansible.windows: "2.3.0"/' galaxy.yml
sed -i 's/ansible\.netcommon:\s*"\*"/ansible.netcommon: "6.1.1"/' galaxy.yml
sed -i 's/ansible\.posix:\s*"\*"/ansible.posix: "1.5.4"/' galaxy.yml
sed -i 's/community\.general:\s*"\*"/community.general: "8.6.0"/' galaxy.yml
sed -i 's/community\.mysql:\s*"\*"/community.mysql: "3.9.0"/' galaxy.yml
sed -i 's/community\.postgresql:\s*"\*"/community.postgresql: "3.4.0"/' galaxy.yml

collection_file=$( basename $(ansible-galaxy collection build -f | awk -F" " '{print $NF}'))
echo "COLLECTION_FILE=$collection_file" >> $GITHUB_ENV

- name: Install the collection
run: ansible-galaxy collection install $COLLECTION_FILE
run: ansible-galaxy collection install -vvvv $COLLECTION_FILE

- name: Run server role tests
run: >-
Expand Down
10 changes: 9 additions & 1 deletion .github/workflows/web.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,11 +68,19 @@ jobs:

- name: Build the collection
run: |
# Pin versions to speed up CI
sed -i 's/ansible\.windows:\s*"\*"/ansible.windows: "2.3.0"/' galaxy.yml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do this? In my mind we should be testing against what we publish in the galaxy.yml file otherwise we may have an issue of experiencing/not experiencing failures that we should/shouldn't during CI/CD testing. Not to mention having to change versions in 5 places instead one 1.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part where galaxy looks up every single version, for every single dependency where we use * takes multiple minutes, multiply that by the matrix of configurations we're testing in github workflows, you're looking at a couple hours just to resolve these dependencies. (Yes, it's really that stupid, I've provided links to reported issues in ce6b2da)

I think it's better we pin this for CI only, as a user could have installed a different collection that pins postgresql to X.Y.Z and if we're fine with that (given we set it to *), galaxy will say requirements are already met and leave it alone anyway, right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinning in CI is not ideal, and especially since it's in this many places, but pinning in galaxy.yml would be the worse of two options.

In any case we should probably specify minimum requirements for all the * dependencies though.

sed -i 's/ansible\.netcommon:\s*"\*"/ansible.netcommon: "6.1.1"/' galaxy.yml
sed -i 's/ansible\.posix:\s*"\*"/ansible.posix: "1.5.4"/' galaxy.yml
sed -i 's/community\.general:\s*"\*"/community.general: "8.6.0"/' galaxy.yml
sed -i 's/community\.mysql:\s*"\*"/community.mysql: "3.9.0"/' galaxy.yml
sed -i 's/community\.postgresql:\s*"\*"/community.postgresql: "3.4.0"/' galaxy.yml

collection_file=$( basename $(ansible-galaxy collection build -f | awk -F" " '{print $NF}'))
echo "COLLECTION_FILE=$collection_file" >> $GITHUB_ENV

- name: Install the collection
run: ansible-galaxy collection install $COLLECTION_FILE
run: ansible-galaxy collection install -vvvv $COLLECTION_FILE

- name: Run role tests
run: >-
Expand Down
4 changes: 2 additions & 2 deletions molecule/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ ansible==7.0.0
ansible-compat==3.0.0
ansible-core==2.14.2
docker==6.1.3
molecule==4.0.4
molecule-docker==2.1.0
molecule<5
molecule-docker @ git+https://github.com/ansible-community/molecule-docker@main
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to change the source to straight to github?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there's a fix that went in after 2.1.0, cb1c1c31a4a468fb243fc2000656504a3ca00bb3
that fixes a bug where the molecule.yml would get overwritten during image building. (I introduced the simple image-building to get python3 on centos7. But I think it's a good way for us to always stay current on all distribution images, instead of the geerlingguy images)

Copy link
Collaborator Author

@eb4x eb4x May 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, the repo is deprecated/archived, so there won't come any further changes to it, but I could pin it to the hash instead?

We should probably go over to molecule 5, but I'd rather get these refactors in before overhauling the new molecule version.

netaddr==1.2.1
pytest==7.2.1
pytest-testinfra==7.0.0
Expand Down
2 changes: 1 addition & 1 deletion molecule/zabbix_javagateway/molecule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ driver:
name: docker

platforms:
- name: zabbix-server-${MY_MOLECULE_CONTAINER:-centos}
- name: zabbix-server-${MY_MOLECULE_VERSION:-v64}-${MY_MOLECULE_DATABASE:-mysql}-${MY_MOLECULE_CONTAINER:-rockylinux8}
image: geerlingguy/docker-${MY_MOLECULE_IMAGE:-rockylinux8}-ansible:latest
privileged: true
pre_build_image: true
Expand Down
6 changes: 2 additions & 4 deletions molecule/zabbix_proxy/destroy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,7 @@

- name: Destroy 3rd party instance(s)
docker_container:
name: '{{ item }}'
name: "{{ item.name }}-db"
state: absent
force_kill: true
with_items:
- mysql-host
- postgresql-host
loop: "{{ molecule_yml.platforms }}"
10 changes: 5 additions & 5 deletions molecule/zabbix_proxy/molecule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
driver:
name: docker
platforms:
- name: zabbix-proxy-${MY_MOLECULE_CONTAINER:-centos}
- name: zabbix-proxy-${MY_MOLECULE_VERSION:-v64}-${MY_MOLECULE_DATABASE:-mysql}-${MY_MOLECULE_CONTAINER:-rockylinux8}
image: geerlingguy/docker-${MY_MOLECULE_IMAGE:-rockylinux8}-ansible:latest
privileged: true
pre_build_image: true
Expand Down Expand Up @@ -37,19 +37,19 @@ provisioner:
zabbix_proxy_dbuser: zabbix-dbuser
zabbix_proxy_database: mysql
zabbix_proxy_dbport: 3306
zabbix_proxy_dbhost: mysql-host
zabbix_proxy_dbhost: "{{ inventory_hostname }}-db"
zabbix_proxy_dbhost_run_install: false
zabbix_proxy_privileged_host: "%"
zabbix_proxy_mysql_login_host: mysql-host
zabbix_proxy_mysql_login_host: "{{ inventory_hostname }}-db"
zabbix_proxy_mysql_login_user: root
zabbix_proxy_mysql_login_password: changeme
zabbix_proxy_mysql_login_port: 3306
pgsql:
zabbix_proxy_database: pgsql
zabbix_proxy_dbport: 5432
zabbix_proxy_dbhost: postgresql-host
zabbix_proxy_dbhost: "{{ inventory_hostname }}-db"
zabbix_proxy_dbhost_run_install: false
zabbix_proxy_pgsql_login_host: postgresql-host
zabbix_proxy_pgsql_login_host: "{{ inventory_hostname }}-db"
zabbix_proxy_pgsql_login_user: postgres
zabbix_proxy_pgsql_login_password: changeme
zabbix_proxy_pgsql_login_port: 5432
Expand Down
4 changes: 2 additions & 2 deletions molecule/zabbix_proxy/prepare.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
pre_tasks:
- name: "Create MySQL Container"
docker_container:
name: mysql-host
name: "{{ item.name }}-db"
image: mysql:8.0
state: started
recreate: true
Expand All @@ -20,7 +20,7 @@

- name: "Create postgresql Container"
docker_container:
name: postgresql-host
name: "{{ item.name }}-db"
image: postgres:13
state: started
recreate: true
Expand Down
6 changes: 2 additions & 4 deletions molecule/zabbix_server/destroy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,7 @@

- name: Destroy 3rd party instance(s)
docker_container:
name: '{{ item }}'
name: "{{ item.name }}-db"
state: absent
force_kill: true
with_items:
- mysql-host
- postgresql-host
loop: "{{ molecule_yml.platforms }}"
10 changes: 5 additions & 5 deletions molecule/zabbix_server/molecule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
driver:
name: docker
platforms:
- name: zabbix-server-${MY_MOLECULE_CONTAINER:-centos}
- name: zabbix-server-${MY_MOLECULE_VERSION:-v64}-${MY_MOLECULE_DATABASE:-mysql}-${MY_MOLECULE_CONTAINER:-rockylinux8}
image: geerlingguy/docker-${MY_MOLECULE_IMAGE:-rockylinux8}-ansible:latest
privileged: true
pre_build_image: true
Expand Down Expand Up @@ -40,19 +40,19 @@ provisioner:
zabbix_server_dbuser: zabbix-dbuser
zabbix_server_database: mysql
zabbix_server_dbport: 3306
zabbix_server_dbhost: mysql-host
zabbix_server_dbhost: "{{ inventory_hostname }}-db"
zabbix_server_dbhost_run_install: false
zabbix_server_privileged_host: "%"
zabbix_server_mysql_login_host: mysql-host
zabbix_server_mysql_login_host: "{{ inventory_hostname }}-db"
zabbix_server_mysql_login_user: root
zabbix_server_mysql_login_password: changeme
zabbix_server_mysql_login_port: 3306
pgsql:
zabbix_server_database: pgsql
zabbix_server_dbport: 5432
zabbix_server_dbhost: postgresql-host
zabbix_server_dbhost: "{{ inventory_hostname }}-db"
zabbix_server_dbhost_run_install: false
zabbix_server_pgsql_login_host: postgresql-host
zabbix_server_pgsql_login_host: "{{ inventory_hostname }}-db"
zabbix_server_pgsql_login_user: postgres
zabbix_server_pgsql_login_password: changeme
zabbix_server_pgsql_login_port: 5432
Expand Down
4 changes: 2 additions & 2 deletions molecule/zabbix_server/prepare.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
pre_tasks:
- name: "Create MySQL Container"
docker_container:
name: mysql-host
name: "{{ item.name }}-db"
image: mysql:8.0.32
state: started
recreate: true
Expand All @@ -20,7 +20,7 @@

- name: "Create postgresql Container"
docker_container:
name: postgresql-host
name: "{{ item.name }}-db"
image: postgres:13
state: started
recreate: true
Expand Down
6 changes: 2 additions & 4 deletions molecule/zabbix_web/destroy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,7 @@

- name: Destroy 3rd party instance(s)
docker_container:
name: '{{ item }}'
name: "{{ item.name }}-db"
state: absent
force_kill: true
with_items:
- mysql-host
- postgresql-host
loop: "{{ molecule_yml.platforms }}"
12 changes: 6 additions & 6 deletions molecule/zabbix_web/molecule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dependency:
driver:
name: docker
platforms:
- name: zabbix-web-${MY_MOLECULE_CONTAINER:-centos}
- name: zabbix-web-${MY_MOLECULE_VERSION:-v64}-${MY_MOLECULE_DATABASE:-mysql}-${MY_MOLECULE_CONTAINER:-rockylinux8}
image: geerlingguy/docker-${MY_MOLECULE_IMAGE:-rockylinux8}-ansible:latest
privileged: true
pre_build_image: true
Expand All @@ -33,7 +33,7 @@ provisioner:
inventory:
group_vars:
all:
zabbix_api_server_url: zabbix-web-${MY_MOLECULE_CONTAINER:-centos}
zabbix_api_server_url: zabbix-web-${MY_MOLECULE_VERSION:-v64}-${MY_MOLECULE_DATABASE:-mysql}-${MY_MOLECULE_CONTAINER:-rockylinux8}
python3:
ansible_python_interpreter: /usr/bin/python3
python:
Expand All @@ -50,19 +50,19 @@ provisioner:
mysql:
zabbix_server_database: mysql
zabbix_server_dbport: 3306
zabbix_server_dbhost: mysql-host
zabbix_server_dbhost: "{{ inventory_hostname }}-db"
zabbix_server_dbhost_run_install: false
zabbix_server_privileged_host: "%"
zabbix_server_mysql_login_host: mysql-host
zabbix_server_mysql_login_host: "{{ inventory_hostname }}-db"
zabbix_server_mysql_login_user: root
zabbix_server_mysql_login_password: changeme
zabbix_server_mysql_login_port: 3306
pgsql:
zabbix_server_database: pgsql
zabbix_server_dbport: 5432
zabbix_server_dbhost: postgresql-host
zabbix_server_dbhost: "{{ inventory_hostname }}-db"
zabbix_server_dbhost_run_install: false
zabbix_server_pgsql_login_host: postgresql-host
zabbix_server_pgsql_login_host: "{{ inventory_hostname }}-db"
zabbix_server_pgsql_login_user: postgres
zabbix_server_pgsql_login_password: changeme
zabbix_server_pgsql_login_port: 5432
Expand Down
4 changes: 2 additions & 2 deletions molecule/zabbix_web/prepare.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
pre_tasks:
- name: "Create MySQL Container"
docker_container:
name: mysql-host
name: "{{ item.name }}-db"
image: mysql:8.0
state: started
recreate: true
Expand All @@ -20,7 +20,7 @@

- name: "Create postgresql Container"
docker_container:
name: postgresql-host
name: "{{ item.name }}-db"
image: postgres:13
state: started
recreate: true
Expand Down
Loading