Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prometheus custom rules not copying to rules folder #484

Open
cdestouni-fmp opened this issue Dec 3, 2024 · 7 comments · May be fixed by #488
Open

Prometheus custom rules not copying to rules folder #484

cdestouni-fmp opened this issue Dec 3, 2024 · 7 comments · May be fixed by #488

Comments

@cdestouni-fmp
Copy link

cdestouni-fmp commented Dec 3, 2024 •
edited
Loading

Setup

I upgraded to the latest release v0.23.0 and run the Prometheus role to configure Prometheus on a Ubuntu 22 virtual machine. My custom rules are located in prometheus/rules/*yml location and my files all end with the .yml extension as expected by the default variable prometheus_alert_rules_files.

Configuration

I 'm running the role using the below ansible configuration.

ansible [core 2.13.13]
  config file = /home/user/folder/ansible/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/user/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/user/.local/bin/ansible
  python version = 3.8.10 (default, Nov  7 2024, 13:10:47) [GCC 9.4.0]
  jinja version = 3.0.3
  libyaml = True

And the below OS configuration

Distributor ID:	Ubuntu
Description: Ubuntu 20.04.5 LTS
Release: 20.04
Codename: focal

Deploying to the below OS configuration

Distributor ID:	Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

Problem

The Copy custom alerting rule files did not copy over my custom rules to the rules directory on the Prometheus server. There was no feedback, even in verbose mode -vvvv so I assumed the glob working correctly and it thinks there's no files to copy over. After the role completed, the Prometheus server was running but no custom rules were displayed in the UI.

Testing done

  • I tried to override the prometheus_alert_rules_files variable with a new location (In the expected list format). This resulted in an error of not locating the files instead of the original no feedback, which makes me think the location is not the problem.

  • I tested at v0.19.0 before this task was last changed and it works fine and copied the custom rules over as expected.

  • I tried the v0.23.0 task changes but using root permissions on the copy task which did not make a difference. This means it is not a permissions issue.

Conclusion

This leaves me to think the problem is situated in the change below.

Working version (v0.19.0)

with_fileglob: "{{ prometheus_alert_rules_files }}"

Not working version (v0.23.0)

loop: "{{ prometheus_alert_rules_files | map('ansible.builtin.fileglob') | flatten }}"

Please let me know if you can help resolve this.
@gardar
Copy link
Member

gardar commented Dec 3, 2024

Do you by any chance have prometheus_alert_rules_files configured as a string and not a list?
See #454 for further explanation

@cdestouni-fmp
Copy link
Author

cdestouni-fmp commented Dec 3, 2024 •
edited
Loading

I left the variable configured with the default list here. I also additionally tested an override where I had it configured as a list and it did not seem to make a difference in this case.

@gardar
Copy link
Member

gardar commented Dec 3, 2024

Hmm, what does your playbook look like? And can you show me the debug log of the task?

@cdestouni-fmp
Copy link
Author

cdestouni-fmp commented Dec 3, 2024 •
edited
Loading

Playbook

Stored in playbooks -> prometheus -> prometheus.yml

- hosts: prometheus
  remote_user: root
  roles:
    - prometheus.prometheus.prometheus
  tags: [prometheus]
  vars:
    prometheus_version: "{{ prometheus.version }}"
    prometheus_storage_retention: "1w"
    prometheus_db_dir: /var/lib/prometheus
    prometheus_config_flags_extra:
      storage.tsdb.min-block-duration: 2h
      storage.tsdb.max-block-duration: 2h
      web.enable-admin-api:
      storage.tsdb.allow-overlapping-blocks:

    prometheus_alertmanager_config:
      - scheme: http
        static_configs:
          - targets: ["{{ alert_manager_url }}"]

    prometheus_alert_rules: []
    prometheus_alert_relabel_configs: []

    prometheus_external_labels:
      clusterName: "production"

    prometheus_targets:
      linux:
        - targets:
            - {{ exampleHostname }}:{{ port }}
            - {{ exampleHostname }}:{{ port }}

I have the prometheus rules setup like so... (In the new rule format)
playbooks -> prometheus -> files -> rules -> custom-alert.yml

The debugging did not output a single thing unfortunately, I tried it with -vvvv and on that particular task there were no logs.

ansible-playbook  -i inventories/dev playbooks/prometheus/prometheus.yml -vvvv

TASK [Copy custom alerting rule files] *********************************************************************************************************************************************

TASK [Configure prometheus] *********************************************************************************************************************************************
changed: [prometheus.host]

@MajorP93
Copy link

MajorP93 commented Dec 5, 2024

Can confirm.
Same issue for me after upgrading the collection to the latest version.

@MajorP93
Copy link

MajorP93 commented Dec 5, 2024

Found the cause:
https://github.com/prometheus-community/ansible/pull/333/files

In this PR support for rule files with file ending .rules was removed.

This was not mentioned in changelog / release notes.
The PR was mentioned under "minor changes". Breaking change was not mentioned.

@MajorP93
Copy link

MajorP93 commented Dec 5, 2024

Renaming my rules files from .rules to .yml alone did not fix my issue.

1e4e4c3#diff-a405b871a41e1b77e6d93c308fb136fd60b40efa6e85917adbde90a8fb9a0f16R93

This change is also problematic.
After reverting this change (loop: "{{ prometheus_alert_rules_files | map('ansible.builtin.fileglob') | flatten }}" --> with_fileglob: "{{ prometheus_alert_rules_files }}") the prometheus role is working as expected.

In other words: I can confirm the conclusion of @cdestouni-fmp

@MajorP93 MajorP93 linked a pull request Dec 5, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: use with_fileglob for iterating over custom alerting rule files MajorP93/ansible
3 participants
@gardar @MajorP93 @cdestouni-fmp