Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: add airflow_scheduler_up metric #56

Open
greenpau opened this issue Aug 2, 2019 · 3 comments
Open

feature: add airflow_scheduler_up metric #56

greenpau opened this issue Aug 2, 2019 · 3 comments

Comments

@greenpau
Copy link

greenpau commented Aug 2, 2019 •
edited
Loading

Add airflow_scheduler_up metric to see if airflow scheduler is up (1) or down (0)
@elephantum
Copy link
Contributor

Good idea. Any thoughts how to track this?

@greenpau
Copy link
Author

greenpau commented Aug 2, 2019

Good idea. Any thoughts how to track this?

@elephantum , I will submit PR for this one.

greenpau added a commit to greenpau/origin_epoch8_airflow-exporter that referenced this issue Aug 3, 2019
@greenpau
add configuration management and authentication features
d42cd5e 
Resolves: epoch8#54, epoch8#55, epoch8#56

More info:

This commit adds the support for the configuration of the exporter via
`prometheus_exporter` section of Airflow configuration.

The keys with `expose_` prefix enable or disable the exposure of certain class
of metrics. For example, Airflow variables-related metrics are not exposed by
default. However, with `expose_variables` set to `True`, the exporter will start
exposing the metrics. Similarly, by default, the exporter exposes scheduler-related
metrics. However, with `expose_scheduler` set to `False`, the exporter will
not export them.

Additionally, when `expose_config` is being enabled, the exporter will expose
a subset of `core` and `prometheus_exporter` configuration settings.

```
[prometheus_exporter]
auth_enabled = True
auth_token = 4e1dba1c-2b66-4275-b8ae-292ee9665fa1
expose_variables = True
expose_config = False
expose_scheduler = False
```

It is possible to disable the exporter:

```
[prometheus_exporter]
disabled = True
```

When authentication is enabled, the metrics are accessible via:

```bash
curl -v -H 'Authorization: Bearer 4e1dba1c-2b66-4275-b8ae-292ee9665fa1' https://localhost:8443/admin/metrics/
```

Also, when the authentication is enabled, Prometheus scrape job
might look like this:

```yaml
  - job_name: 'airflow_exporters'
    metrics_path: /admin/metrics/
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token: '4e1dba1c-2b66-4275-b8ae-292ee9665fa1'
    scrape_interval: 5m
    static_configs:
    - targets:
      - '127.0.0.1:8443'
      labels:
        environment: 'dev'
    relabel_configs:
    - source_labels: [__address__]
      regex: "^(.*):(.*)$"
      target_label: instance
      replacement: ${1}
    - source_labels: [instance]
      regex: "^(127.0.0.1)$"
      target_label: instance
      replacement: "airflow"
```
@greenpau greenpau mentioned this issue Aug 3, 2019
Closed
@greenpau
Copy link
Author

greenpau commented Aug 3, 2019

@elephantum , see https://travis-ci.org/epoch8/airflow-exporter/jobs/567214466#L463

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@elephantum @greenpau