Skip to content

Commit

Permalink
Merge pull request #21 from luissimas/feature/victoria-metrics-support
Browse files Browse the repository at this point in the history 
Feature/victoria metrics support
  • Loading branch information
@luissimas
luissimas authored Jul 30, 2024
2 parents 44b53a3 + 7be38de commit 909bcce
Show file tree
Hide file tree
Showing 16 changed files with 1,605 additions and 51 deletions.
45 changes: 25 additions & 20 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

# Zettelkasten exporter

An agent that collects metrics from an zettelkasten and stores them into an InfluxDB bucket.
An agent that collects metrics from an zettelkasten and lets you visualize them in Grafana.

![](./docs/assets/dashboard.png)

Expand All @@ -18,6 +18,7 @@ An agent that collects metrics from an zettelkasten and stores them into an Infl
- Parses both markdown and wiki links
- Authenticate in private git repositories using personal access tokens
- Grafana dashboards included
- Support for both InfluxDB and VictoriaMetrics as storage backends

## Usage

Expand All @@ -26,23 +27,29 @@ The exporter is distributed as both a binary and a Docker image. The currently d
- [Docker compose example](./examples/compose)
- [Kubernetes example](./examples/kubernetes)

Note that for a complete solution, it will be necessary to setup InfluxDB and Grafana. For more information about setting them up, refer to their documentation. Here are some links that might be useful:
Note that for a complete solution, it will be necessary to setup Grafana and either InfluxDB or VictoriaMetrics as data sources. For more information about setting them up, refer to their documentation. Here are some links that might be useful:

- https://grafana.com/docs/grafana/latest/getting-started/get-started-grafana-influxdb/
- https://docs.influxdata.com/influxdb/v2/get-started/setup/
- https://docs.victoriametrics.com/

The provided dashboard uses `Flux` as the language to query InfluxDB, so make sure to set the "Query language" option to "Flux" when creating the InfluxDB data source in Grafana.
In the `dashboards` folder there are two Grafana dashboards provided: one for InfluxDB and another for VictoriaMetrics. You'll have to import the appropriate dashboard for your storage backend.

The provided InfluxDB dashboard uses `Flux` as the query language, so make sure to set the "Query language" option to "Flux" when creating the InfluxDB data source in Grafana.

For both storage backends, make sure to configure the data retention period according to your needs.

## Configuration

All configuration is supplied via environment variables. You should supply at least the required variables and either the `ZETTELKASTEN_DIRECTORY` or the `ZETTELKASTEN_GIT_URL` variables.
All configuration is supplied via environment variables. You should supply at least the zettelkasten source via the `ZETTELKASTEN_DIRECTORY` or `ZETTELKASTEN_GIT_URL` variables and the storage backend via the `VICTORIAMETRICS_URL` or `INFLUXDB_*` variables.

| Name | Description | Default | Required |
| -------------------------- | -------------------------------------------------------------------- | ------------------------------ | -------- |
| INFLUXDB_URL | The InfluxDB URL | | Yes |
| INFLUXDB_TOKEN | The InfluxDB token to authenticate in the bucket | | Yes |
| INFLUXDB_ORG | The InfluxDB org containing the bucket | | Yes |
| INFLUXDB_BUCKET | The InfluxDB bucket to register metrics | | Yes |
| VICTORIAMETRICS_URL | The VictoriaMetrics URL | | No |
| INFLUXDB_URL | The InfluxDB URL | | No |
| INFLUXDB_TOKEN | The InfluxDB token to authenticate in the bucket | | No |
| INFLUXDB_ORG | The InfluxDB org containing the bucket | | No |
| INFLUXDB_BUCKET | The InfluxDB bucket to register metrics | | No |
| ZETTELKASTEN_DIRECTORY | The local directory containing the zettelkasten | | No |
| ZETTELKASTEN_GIT_URL | The URL for the git repository containing the zettelkasten | | No |
| ZETTELKASTEN_GIT_TOKEN | The access token to authenticate with private repositories | | No |
Expand All @@ -54,26 +61,24 @@ All configuration is supplied via environment variables. You should supply at le

## Metrics

The exporter collects metrics by parsing the contents of the markdown files present in the Zettelkasten. Currently the exporter stores metrics for individual notes and also aggregated metrics describing the entire Zettelkasten. The combination of raw and pre processed metrics allows for both flexibility and efficiency when querying the data, at the cost of a slightly higher storage usage. The two sets of metrics are stored in the same InfluxDB bucket under different [measurement names](https://docs.influxdata.com/influxdb/cloud/reference/key-concepts/data-elements/#measurement).
The exporter collects metrics by parsing the contents of the markdown files present in the Zettelkasten. Currently the exporter stores metrics for individual notes and also aggregated metrics describing the entire Zettelkasten. The combination of raw and pre processed metrics allows for both flexibility and efficiency when querying the data, at the cost of a slightly higher storage usage. When using the InfluxDB storage, the two sets of metrics are stored in the same InfluxDB bucket under different [measurement names](https://docs.influxdata.com/influxdb/cloud/reference/key-concepts/data-elements/#measurement). When using the VictoriaMetrics storage, each metric is stored under a different name.

The following table describes all metrics collected by the exporter and their respective measurement names:

| Measurement | Name | Description |
|-------------|----------------|-----------------------------------------|
| notes | link_count | Number of links in the note |
| notes | word_count | Number of words in the note |
| notes | backlink_count | Number of links that reference the note |
| total | note_count | Number of notes in the Zettelkasten |
| total | link_count | Number of links in the Zettelkasten |
| total | word_count | Number of words in the Zettelkasten |
| InfluxDB measurement | InfluxDB name | VictoriaMetrics name | Description |
|----------------------|----------------|----------------------|-----------------------------------------|
| notes | link_count | notes_link_count | Number of links in the note |
| notes | word_count | notes_word_count | Number of words in the note |
| notes | backlink_count | notes_backlink_count | Number of links that reference the note |
| total | note_count | total_note_count | Number of notes in the Zettelkasten |
| total | link_count | total_link_count | Number of links in the Zettelkasten |
| total | word_count | total_word_count | Number of words in the Zettelkasten |

## Roadmap

These are some features that I'd like to include in the future.

- Collect additional metrics
- Support Prometheus remote write as a storage
- Support VictoriaMetrics as a storage
- Support Prometheus remote write as a storage backend

## References

Expand Down
95 changes: 86 additions & 9 deletions dashboards/Zettelkasten-InfluxDB.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 2,
"id": 1,
"id": 67,
"links": [],
"panels": [
{
Expand Down Expand Up @@ -82,7 +82,7 @@
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.0.1",
"pluginVersion": "10.4.1",
"targets": [
{
"datasource": {
Expand Down Expand Up @@ -144,7 +144,7 @@
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.0.1",
"pluginVersion": "10.4.1",
"targets": [
{
"datasource": {
Expand Down Expand Up @@ -206,7 +206,7 @@
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.0.1",
"pluginVersion": "10.4.1",
"targets": [
{
"datasource": {
Expand Down Expand Up @@ -269,7 +269,7 @@
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.0.1",
"pluginVersion": "10.4.1",
"targets": [
{
"datasource": {
Expand Down Expand Up @@ -355,7 +355,7 @@
}
]
},
"pluginVersion": "11.0.1",
"pluginVersion": "10.4.1",
"targets": [
{
"datasource": {
Expand Down Expand Up @@ -502,6 +502,19 @@
}
],
"title": "Notes",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {},
"includeByName": {},
"indexByName": {},
"renameByName": {
"note_count": "Note count"
}
}
}
],
"type": "timeseries"
},
{
Expand Down Expand Up @@ -592,6 +605,19 @@
}
],
"title": "Words",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {},
"includeByName": {},
"indexByName": {},
"renameByName": {
"word_count": "Word count"
}
}
}
],
"type": "timeseries"
},
{
Expand Down Expand Up @@ -683,6 +709,19 @@
}
],
"title": "Links",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {},
"includeByName": {},
"indexByName": {},
"renameByName": {
"link_count": "Link count"
}
}
}
],
"type": "timeseries"
},
{
Expand Down Expand Up @@ -774,6 +813,19 @@
}
],
"title": "Reading time",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {},
"includeByName": {},
"indexByName": {},
"renameByName": {
"word_count": "Reading time"
}
}
}
],
"type": "timeseries"
},
{
Expand Down Expand Up @@ -865,6 +917,19 @@
}
],
"title": "Average link count",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {},
"includeByName": {},
"indexByName": {},
"renameByName": {
"Value": "Average link count"
}
}
}
],
"type": "timeseries"
},
{
Expand Down Expand Up @@ -956,6 +1021,19 @@
}
],
"title": "Average word count",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {},
"includeByName": {},
"indexByName": {},
"renameByName": {
"Value": "Average word count"
}
}
}
],
"type": "timeseries"
}
],
Expand All @@ -968,7 +1046,7 @@
"current": {
"selected": false,
"text": "influxdb",
"value": "cdqmo6y19tc74d"
"value": "cdpk3wq4tof7kc"
},
"hide": 0,
"includeAll": false,
Expand All @@ -988,11 +1066,10 @@
"from": "now-30d",
"to": "now"
},
"timeRangeUpdatedDuringEditOrView": false,
"timepicker": {},
"timezone": "browser",
"title": "Zettelkasten",
"uid": "fdoghlpqzr5kwe",
"version": 11,
"version": 2,
"weekStart": ""
}
Loading

0 comments on commit 909bcce

Please sign in to comment.