Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

open_metrics not in telemetry path although seen scraped in logs #116

Open
jcocozzi opened this issue Nov 1, 2023 · 22 comments
Open

open_metrics not in telemetry path although seen scraped in logs #116

jcocozzi opened this issue Nov 1, 2023 · 22 comments
Labels
keep-open Make stale action ignore the Issue/PR Need Investigation

Comments

@jcocozzi
Copy link

jcocozzi commented Nov 1, 2023

Overview of the Issue

Scrapes of open_metrics are not seen on telemetry_path when running canary with open_metrics supplied as an optional metric.

Reproduction Steps

Configure artifactory exporter with optional metric set to open_metrics, debug logging, and deploy to k8s cluster.

Verify open metrics are getting scraped in the pod log. With debug logging you will see the log entry level=debug ts=2023-11-01T14:56:09.610Z caller=openmetrics.go:26 msg="OpenMetrics from Artifactory" followed by all the open_metrics.

Set up port forwarding on prometheus scrape port.

Curl for http://localhost:<scrape_port>/metrics.

Various default metrics will be seen, but no open_metrics.

Operating system and Environment details

Canary version deployed on k8s cluster using helm chart found here: https://github.com/peimanja/helm-charts/tree/main/charts/prometheus-artifactory-exporter

Logs

FROM POD LOG:

level=debug ts=2023-11-01T14:56:09.610Z caller=openmetrics.go:26 msg="OpenMetrics from Artifactory" body="# HELP jfsh_binaries_download_total Counts the total binaries download\n# UPDATED jfsh_binaries_download_total 1698850564126\n# TYPE jfsh_binaries_download_total counter\njfsh_binaries_download_total{id="cache-fs",name="cache_fs"} 10592 1698850569605\n# HELP jfsh_binaries_not_found_total Counts the total request not found\n# UPDATED jfsh_binaries_not_found_total 1698850564126\n# TYPE jfsh_binaries_not_found_total counter\njfsh_binaries_not_found_total{id="cache-fs",name="cache_fs"} 2698 1698850569605\n# HELP jfsh_cache_evicted_bytes Counts the amount of cleared bytes from cache-fs\n# UPDATED jfsh_cache_evicted_bytes 1698850564126\n# TYPE jfsh_cache_evicted_bytes gauge\njfsh_cache_evicted_bytes{id="cache-fs",name="cache_fs"} 0 1698850569605\n# HELP jfsh_storage_size_bytes Size (bytes) in storage\n# UPDATED jfsh_storage_size_bytes 1698850564126\n# TYPE jfsh_storage_size_bytes gauge\njfsh_storage_size_bytes{id="cache-fs",name="cache_fs"} 6113447163 1698850569605\n# HELP jfsh_binaries_count_total Number of binaries in storage\n# UPDATED jfsh_binaries_count_total 1698850564126\n# TYPE jfsh_binaries_count_total gauge\njfsh_binaries_count_total{id="cache-fs",name="cache_fs"} 2711 1698850569605\n# HELP jfsh_binaries_download_total Counts the total binaries download\n# UPDATED jfsh_binaries_download_total 1698850564126\n# TYPE jfsh_binaries_download_total counter\njfsh_binaries_download_total{id="s3-storage-v3",name="s3_storage_v3"} 2583 1698850569605\n# HELP jfsh_binaries_upload_total Counts binaries uploads\n# UPDATED jfsh_binaries_upload_total 1698850564126\n# TYPE jfsh_binaries_upload_total counter\njfsh_binaries_upload_total{id="s3-storage-v3",name="s3_storage_v3"} 2066 1698850569605\n# HELP jfsh_signed_url_redirect_total Counts how many times signed url was generated\n# UPDATED jfsh_signed_url_redirect_total 1698850564126\n# TYPE jfsh_signed_url_redirect_total counter\njfsh_signed_url_redirect_total{id="s3-storage-v3",name="s3_storage_v3"} 0 1698850569605\n# HELP jfrt_db_connections_active_total Total Active Connections\n# UPDATED jfrt_db_connections_active_total 1698850564126\n# TYPE jfrt_db_connections_active_total gauge\njfrt_db_connections_active_total 0 1698850569605\n# HELP jfrt_db_connections_idle_total Total Idle Connections\n# UPDATED jfrt_db_connections_idle_total 1698850564126\n# TYPE jfrt_db_connections_idle_total gauge\njfrt_db_connections_idle_total 2 1698850569605\n# HELP jfrt_db_connections_max_active_total Total Max Active Connections\n# UPDATED jfrt_db_connections_max_active_total 1698850564126\n# TYPE jfrt_db_connections_max_active_total gauge\njfrt_db_connections_max_active_total 300 1698850569605\n# HELP jfrt_db_connections_min_idle_total Total Min Idle Connections\n# UPDATED jfrt_db_connections_min_idle_total 1698850564126\n# TYPE jfrt_db_connections_min_idle_total gauge\njfrt_db_connections_min_idle_total 1 1698850569605\n# HELP app_disk_used_bytes Used bytes for app home directory disk device\n# UPDATED app_disk_used_bytes 1698850569605\n# TYPE app_disk_used_bytes gauge\napp_disk_used_bytes 79492304896 1698850569605\n# HELP app_disk_free_bytes Free bytes for app home directory disk device\n# UPDATED app_disk_free_bytes 1698850569605\n# TYPE app_disk_free_bytes gauge\napp_disk_free_bytes 130749255680 1698850569605\n# HELP adoption_artifactory_authentication_count Authentication\n# UPDATED adoption_artifactory_authentication_count 1698850569605\n# TYPE adoption_artifactory_authentication_count counter\nadoption_artifactory_authentication_count{action="login",apiKey="false",domain="adoption",realm="internal",token="true"} 33 1698850569605\n# HELP adoption_artifactory_authentication_count Authentication\n# UPDATED adoption_artifactory_authentication_count 1698850569605\n# TYPE adoption_artifactory_authentication_count counter\nadoption_artifactory_authentication_count{action="login",apiKey="true",domain="adoption",realm="oauth",token="false"} 1 1698850569605\n# HELP adoption_artifactory_authentication_count Authentication\n# UPDATED adoption_artifactory_authentication_count 1698850569605\n# TYPE adoption_artifactory_authentication_count counter\nadoption_artifactory_authentication_count{action="login",apiKey="false",domain="adoption",realm="oauth",token="true"} 8779 1698850569605\n# HELP jfsh_thread_pool_thread_pool_active_count_total Number of threads that are actively executing tasks\n# UPDATED jfsh_thread_pool_thread_pool_active_count_total 1698850564126\n# TYPE jfsh_thread_pool_thread_pool_active_count_total gauge\njfsh_thread_pool_thread_pool_active_count_total{maxPoolSize="3.0",mergedPools="0",name="binary-store-gc"} 0.0 1698850569605\n# HELP jfsh_thread_pool_thread_pool_pending_tasks_total Number of tasks pending execution\n# UPDATED jfsh_thread_pool_thread_pool_pending_tasks_total 1698850564126\n# TYPE jfsh_thread_pool_thread_pool_pending_tasks_total gauge\njfsh_thread_pool_thread_pool_pending_tasks_total{maxPoolSize="3.0",mergedPools="0",name="binary-store-gc"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_created_seconds RB V2 version creation time: percentile 50%\n# UPDATED jfbh_rbv2_version_created_seconds 1698850564127\n# TYPE jfbh_rbv2_version_created_seconds gauge\njfbh_rbv2_version_created_seconds{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_created_seconds RB V2 version creation time: percentile 75%\n# UPDATED jfbh_rbv2_version_created_seconds 1698850564127\n# TYPE jfbh_rbv2_version_created_seconds gauge\njfbh_rbv2_version_created_seconds{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_created_seconds RB V2 version creation time: percentile 95%\n# UPDATED jfbh_rbv2_version_created_seconds 1698850564127\n# TYPE jfbh_rbv2_version_created_seconds gauge\njfbh_rbv2_version_created_seconds{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_created_seconds RB V2 version creation time: percentile 99%\n# UPDATED jfbh_rbv2_version_created_seconds 1698850564127\n# TYPE jfbh_rbv2_version_created_seconds gauge\njfbh_rbv2_version_created_seconds{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_created_seconds RB V2 version creation time: standard deviation\n# UPDATED jfbh_rbv2_version_created_seconds 1698850564127\n# TYPE jfbh_rbv2_version_created_seconds gauge\njfbh_rbv2_version_created_seconds{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_created_seconds RB V2 promotion creation time: percentile 50%\n# UPDATED jfbh_rbv2_promotion_created_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_created_seconds gauge\njfbh_rbv2_promotion_created_seconds{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_created_seconds RB V2 promotion creation time: percentile 75%\n# UPDATED jfbh_rbv2_promotion_created_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_created_seconds gauge\njfbh_rbv2_promotion_created_seconds{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_created_seconds RB V2 promotion creation time: percentile 95%\n# UPDATED jfbh_rbv2_promotion_created_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_created_seconds gauge\njfbh_rbv2_promotion_created_seconds{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_created_seconds RB V2 promotion creation time: percentile 99%\n# UPDATED jfbh_rbv2_promotion_created_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_created_seconds gauge\njfbh_rbv2_promotion_created_seconds{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_created_seconds RB V2 promotion creation time: standard deviation\n# UPDATED jfbh_rbv2_promotion_created_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_created_seconds gauge\njfbh_rbv2_promotion_created_seconds{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_deleted_seconds RB V2 promotion deletion time: percentile 50%\n# UPDATED jfbh_rbv2_promotion_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_deleted_seconds gauge\njfbh_rbv2_promotion_deleted_seconds{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_deleted_seconds RB V2 promotion deletion time: percentile 75%\n# UPDATED jfbh_rbv2_promotion_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_deleted_seconds gauge\njfbh_rbv2_promotion_deleted_seconds{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_deleted_seconds RB V2 promotion deletion time: percentile 95%\n# UPDATED jfbh_rbv2_promotion_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_deleted_seconds gauge\njfbh_rbv2_promotion_deleted_seconds{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_deleted_seconds RB V2 promotion deletion time: percentile 99%\n# UPDATED jfbh_rbv2_promotion_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_deleted_seconds gauge\njfbh_rbv2_promotion_deleted_seconds{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_deleted_seconds RB V2 promotion deletion time: standard deviation\n# UPDATED jfbh_rbv2_promotion_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_deleted_seconds gauge\njfbh_rbv2_promotion_deleted_seconds{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_deleted_seconds RB V2 version deletion time: percentile 50%\n# UPDATED jfbh_rbv2_version_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_version_deleted_seconds gauge\njfbh_rbv2_version_deleted_seconds{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_deleted_seconds RB V2 version deletion time: percentile 75%\n# UPDATED jfbh_rbv2_version_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_version_deleted_seconds gauge\njfbh_rbv2_version_deleted_seconds{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_deleted_seconds RB V2 version deletion time: percentile 95%\n# UPDATED jfbh_rbv2_version_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_version_deleted_seconds gauge\njfbh_rbv2_version_deleted_seconds{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_deleted_seconds RB V2 version deletion time: percentile 99%\n# UPDATED jfbh_rbv2_version_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_version_deleted_seconds gauge\njfbh_rbv2_version_deleted_seconds{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_deleted_seconds RB V2 version deletion time: standard deviation\n# UPDATED jfbh_rbv2_version_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_version_deleted_seconds gauge\njfbh_rbv2_version_deleted_seconds{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifact_copied_seconds RB V2 artifact copy time: percentile 50%\n# UPDATED jfbh_rbv2_artifact_copied_seconds 1698850564127\n# TYPE jfbh_rbv2_artifact_copied_seconds gauge\njfbh_rbv2_artifact_copied_seconds{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifact_copied_seconds RB V2 artifact copy time: percentile 75%\n# UPDATED jfbh_rbv2_artifact_copied_seconds 1698850564127\n# TYPE jfbh_rbv2_artifact_copied_seconds gauge\njfbh_rbv2_artifact_copied_seconds{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifact_copied_seconds RB V2 artifact copy time: percentile 95%\n# UPDATED jfbh_rbv2_artifact_copied_seconds 1698850564127\n# TYPE jfbh_rbv2_artifact_copied_seconds gauge\njfbh_rbv2_artifact_copied_seconds{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifact_copied_seconds RB V2 artifact copy time: percentile 99%\n# UPDATED jfbh_rbv2_artifact_copied_seconds 1698850564127\n# TYPE jfbh_rbv2_artifact_copied_seconds gauge\njfbh_rbv2_artifact_copied_seconds{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifact_copied_seconds RB V2 artifact copy time: standard deviation\n# UPDATED jfbh_rbv2_artifact_copied_seconds 1698850564127\n# TYPE jfbh_rbv2_artifact_copied_seconds gauge\njfbh_rbv2_artifact_copied_seconds{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifacts_total RB V2 artifacts: percentile 50%\n# UPDATED jfbh_rbv2_artifacts_total 1698850564127\n# TYPE jfbh_rbv2_artifacts_total gauge\njfbh_rbv2_artifacts_total{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifacts_total RB V2 artifacts: percentile 75%\n# UPDATED jfbh_rbv2_artifacts_total 1698850564127\n# TYPE jfbh_rbv2_artifacts_total gauge\njfbh_rbv2_artifacts_total{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifacts_total RB V2 artifacts: percentile 95%\n# UPDATED jfbh_rbv2_artifacts_total 1698850564127\n# TYPE jfbh_rbv2_artifacts_total gauge\njfbh_rbv2_artifacts_total{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifacts_total RB V2 artifacts: percentile 99%\n# UPDATED jfbh_rbv2_artifacts_total 1698850564127\n# TYPE jfbh_rbv2_artifacts_total gauge\njfbh_rbv2_artifacts_total{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifacts_total RB V2 artifacts: standard deviation\n# UPDATED jfbh_rbv2_artifacts_total 1698850564127\n# TYPE jfbh_rbv2_artifacts_total gauge\njfbh_rbv2_artifacts_total{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_properties_total RB V2 properties: percentile 50%\n# UPDATED jfbh_rbv2_properties_total 1698850564127\n# TYPE jfbh_rbv2_properties_total gauge\njfbh_rbv2_properties_total{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_properties_total RB V2 properties: percentile 75%\n# UPDATED jfbh_rbv2_properties_total 1698850564127\n# TYPE jfbh_rbv2_properties_total gauge\njfbh_rbv2_properties_total{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_properties_total RB V2 properties: percentile 95%\n# UPDATED jfbh_rbv2_properties_total 1698850564127\n# TYPE jfbh_rbv2_properties_total gauge\njfbh_rbv2_properties_total{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_properties_total RB V2 properties: percentile 99%\n# UPDATED jfbh_rbv2_properties_total 1698850564127\n# TYPE jfbh_rbv2_properties_total gauge\njfbh_rbv2_properties_total{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_properties_total RB V2 properties: standard deviation\n# UPDATED jfbh_rbv2_properties_total 1698850564127\n# TYPE jfbh_rbv2_properties_total gauge\njfbh_rbv2_properties_total{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_creation_methods_total RB V2 creation methods\n# UPDATED jfbh_rbv2_creation_methods_total 1698850564127\n# TYPE jfbh_rbv2_creation_methods_total counter\njfbh_rbv2_creation_methods_total{source_method="artifacts"} 0 1698850569605\n# HELP jfbh_rbv2_creation_methods_total RB V2 creation methods\n# UPDATED jfbh_rbv2_creation_methods_total 1698850564127\n# TYPE jfbh_rbv2_creation_methods_total counter\njfbh_rbv2_creation_methods_total{source_method="release_bundles"} 0 1698850569605\n# HELP jfbh_rbv2_creation_methods_total RB V2 creation methods\n# UPDATED jfbh_rbv2_creation_methods_total 1698850564127\n# TYPE jfbh_rbv2_creation_methods_total counter\njfbh_rbv2_creation_methods_total{source_method="aql"} 0 1698850569605\n# HELP jfbh_rbv2_creation_methods_total RB V2 creation methods\n# UPDATED jfbh_rbv2_creation_methods_total 1698850564127\n# TYPE jfbh_rbv2_creation_methods_total counter\njfbh_rbv2_creation_methods_total{source_method="builds"} 0 1698850569605\n# HELP jfbh_rbv2_creation_agents_total RB V2 creation agents\n# UPDATED jfbh_rbv2_creation_agents_total 1698850564127\n# TYPE jfbh_rbv2_creation_agents_total counter\njfbh_rbv2_creation_agents_total{source_agent="api"} 0 1698850569605\n# HELP jfbh_rbv2_creation_agents_total RB V2 creation agents\n# UPDATED jfbh_rbv2_creation_agents_total 1698850564127\n# TYPE jfbh_rbv2_creation_agents_total counter\njfbh_rbv2_creation_agents_total{source_agent="browser"} 0 1698850569605\n# HELP jfbh_rbv2_creation_agents_total RB V2 creation agents\n# UPDATED jfbh_rbv2_creation_agents_total 1698850564127\n# TYPE jfbh_rbv2_creation_agents_total counter\njfbh_rbv2_creation_agents_total{source_agent="cli"} 0 1698850569605\n# HELP jfbh_rbv2_bundle_operations_total RB V2 bundle operations\n# UPDATED jfbh_rbv2_bundle_operations_total 1698850564127\n# TYPE jfbh_rbv2_bundle_operations_total counter\njfbh_rbv2_bundle_operations_total{operation="create_bundle"} 0 1698850569605\n# HELP jfbh_rbv2_bundle_operations_total RB V2 bundle operations\n# UPDATED jfbh_rbv2_bundle_operations_total 1698850564127\n# TYPE jfbh_rbv2_bundle_operations_total counter\njfbh_rbv2_bundle_operations_total{operation="delete_bundle"} 0 1698850569605\n# HELP jfbh_rbv2_promotion_operations_total RB V2 promotion operations\n# UPDATED jfbh_rbv2_promotion_operations_total 1698850564127\n# TYPE jfbh_rbv2_promotion_operations_total counter\njfbh_rbv2_promotion_operations_total{operation="delete_promotion"} 0 1698850569605\n# HELP jfbh_rbv2_promotion_operations_total RB V2 promotion operations\n# UPDATED jfbh_rbv2_promotion_operations_total 1698850564127\n# TYPE jfbh_rbv2_promotion_operations_total counter\njfbh_rbv2_promotion_operations_total{operation="create_promotion"} 0 1698850569605\n# HELP jfbh_rbv2_release_bundles_total RB V2 release bundles\n# UPDATED jfbh_rbv2_release_bundles_total 1698850564127\n# TYPE jfbh_rbv2_release_bundles_total gauge\njfbh_rbv2_release_bundles_total 0 1698850569605\n# HELP jfbh_rbv2_release_bundles_completed_total RB V2 release bundles completed\n# UPDATED jfbh_rbv2_release_bundles_completed_total 1698850564127\n# TYPE jfbh_rbv2_release_bundles_completed_total gauge\njfbh_rbv2_release_bundles_completed_total 0 1698850569605\n# HELP jfbh_rbv2_release_bundles_failed_total RB V2 release bundles failed\n# UPDATED jfbh_rbv2_release_bundles_failed_total 1698850564127\n# TYPE jfbh_rbv2_release_bundles_failed_total gauge\njfbh_rbv2_release_bundles_failed_total 0 1698850569605\n# HELP jfbh_rbv2_promotions_total RB V2 promotions\n# UPDATED jfbh_rbv2_promotions_total 1698850564127\n# TYPE jfbh_rbv2_promotions_total gauge\njfbh_rbv2_promotions_total 0 1698850569605\n# HELP jfbh_rbv2_promotions_completed_total RB V2 promotions completed\n# UPDATED jfbh_rbv2_promotions_completed_total 1698850564127\n# TYPE jfbh_rbv2_promotions_completed_total gauge\njfbh_rbv2_promotions_completed_total 0 1698850569605\n# HELP jfbh_rbv2_promotions_failed_total RB V2 promotions failed\n# UPDATED jfbh_rbv2_promotions_failed_total 1698850564127\n# TYPE jfbh_rbv2_promotions_failed_total gauge\njfbh_rbv2_promotions_failed_total 0 1698850569605\n# HELP jfrt_events_gc_size_cleaned_bytes Total Bytes recovered by Garbage Collection\n# UPDATED jfrt_events_gc_size_cleaned_bytes 1698850564126\n# TYPE jfrt_events_gc_size_cleaned_bytes gauge\njfrt_events_gc_size_cleaned_bytes{type="EVENTS_GC"} 0 1698850569605\n# HELP jfrt_events_gc_binaries_deleted_total Total number of binaries removed by Garbage Collection\n# UPDATED jfrt_events_gc_binaries_deleted_total 1698850564126\n# TYPE jfrt_events_gc_binaries_deleted_total counter\njfrt_events_gc_binaries_deleted_total{type="EVENTS_GC"} 0 1698850569605\n# HELP jfrt_events_gc_waiting_total Total number of waiting event\n# UPDATED jfrt_events_gc_waiting_total 1698850564126\n# TYPE jfrt_events_gc_waiting_total gauge\njfrt_events_gc_waiting_total{type="EVENTS_GC"} 0 1698850569605\n# HELP jfrt_events_gc_events_total Total number of event\n# UPDATED jfrt_events_gc_events_total 1698850564126\n# TYPE jfrt_events_gc_events_total counter\njfrt_events_gc_events_total{type="EVENTS_GC"} 0 1698850569605\n# HELP jfrt_runtime_heap_freememory_bytes Free Memory\n# UPDATED jfrt_runtime_heap_freememory_bytes 1698850569605\n# TYPE jfrt_runtime_heap_freememory_bytes gauge\njfrt_runtime_heap_freememory_bytes 3532722600 1698850569605\n# HELP jfrt_runtime_heap_maxmemory_bytes Max Memory\n# UPDATED jfrt_runtime_heap_maxmemory_bytes 1698850569605\n# TYPE jfrt_runtime_heap_maxmemory_bytes gauge\njfrt_runtime_heap_maxmemory_bytes 12884901888 1698850569605\n# HELP jfrt_runtime_heap_totalmemory_bytes Total Memory\n# UPDATED jfrt_runtime_heap_totalmemory_bytes 1698850569605\n# TYPE jfrt_runtime_heap_totalmemory_bytes gauge\njfrt_runtime_heap_totalmemory_bytes 8589934592 1698850569605\n# HELP jfrt_runtime_heap_processors_total Available Processors\n# UPDATED jfrt_runtime_heap_processors_total 1698850569605\n# TYPE jfrt_runtime_heap_processors_total counter\njfrt_runtime_heap_processors_total 4 1698850569605\n# HELP jfrt_projects_active_total Projects Amount\n# UPDATED jfrt_projects_active_total 1698850569605\n# TYPE jfrt_projects_active_total counter\njfrt_projects_active_total 1 1698850569605\n# HELP jfrt_storage_current_total_size_bytes Used Storage\n# UPDATED jfrt_storage_current_total_size_bytes 1698850569605\n# TYPE jfrt_storage_current_total_size_bytes gauge\njfrt_storage_current_total_size_bytes 0 1698850569605\n# HELP jfrt_artifacts_gc_next_run_seconds Next GC Run\n# UPDATED jfrt_artifacts_gc_next_run_seconds 1698840003794\n# TYPE jfrt_artifacts_gc_next_run_seconds gauge\njfrt_artifacts_gc_next_run_seconds 14396 1698850569605\n# HELP jfrt_artifacts_gc_duration_seconds Time taken for garbage collection\n# UPDATED jfrt_artifacts_gc_duration_seconds 1698840003794\n# TYPE jfrt_artifacts_gc_duration_seconds gauge\njfrt_artifacts_gc_duration_seconds{end="1698840000346",start="1698840000005",status="COMPLETED",type="TRASH_AND_BINARIES"} 0.341 1698850569605\n# HELP jfrt_artifacts_gc_size_cleaned_bytes Total Bytes recovered by Garbage Collection\n# UPDATED jfrt_artifacts_gc_size_cleaned_bytes 1698840003794\n# TYPE jfrt_artifacts_gc_size_cleaned_bytes gauge\njfrt_artifacts_gc_size_cleaned_bytes{end="1698840000346",start="1698840000005",status="COMPLETED",type="TRASH_AND_BINARIES"} 0 1698850569605\n# HELP jfrt_artifacts_gc_binaries_total Total number of binaries removed by Garbage Collection\n# UPDATED jfrt_artifacts_gc_binaries_total 1698840003794\n# TYPE jfrt_artifacts_gc_binaries_total gauge\njfrt_artifacts_gc_binaries_total{end="1698840000346",start="1698840000005",status="COMPLETED",type="TRASH_AND_BINARIES"} 0 1698850569605\n"
level=debug ts=2023-11-01T14:56:09.612Z caller=openMetrics.go:21 msg="OpenMetrics from Artifactory util" body="# HELP jfsh_binaries_download_total Counts the total binaries download\n# UPDATED jfsh_binaries_download_total 1698850564126\n# TYPE jfsh_binaries_download_total counter\njfsh_binaries_download_total{id="cache-fs",name="cache_fs"} 10592 1698850569605\n# HELP jfsh_binaries_not_found_total Counts the total request not found\n# UPDATED jfsh_binaries_not_found_total 1698850564126\n# TYPE jfsh_binaries_not_found_total counter\njfsh_binaries_not_found_total{id="cache-fs",name="cache_fs"} 2698 1698850569605\n# HELP jfsh_cache_evicted_bytes Counts the amount of cleared bytes from cache-fs\n# UPDATED jfsh_cache_evicted_bytes 1698850564126\n# TYPE jfsh_cache_evicted_bytes gauge\njfsh_cache_evicted_bytes{id="cache-fs",name="cache_fs"} 0 1698850569605\n# HELP jfsh_storage_size_bytes Size (bytes) in storage\n# UPDATED jfsh_storage_size_bytes 1698850564126\n# TYPE jfsh_storage_size_bytes gauge\njfsh_storage_size_bytes{id="cache-fs",name="cache_fs"} 6113447163 1698850569605\n# HELP jfsh_binaries_count_total Number of binaries in storage\n# UPDATED jfsh_binaries_count_total 1698850564126\n# TYPE jfsh_binaries_count_total gauge\njfsh_binaries_count_total{id="cache-fs",name="cache_fs"} 2711 1698850569605\n# HELP jfsh_binaries_download_total Counts the total binaries download\n# UPDATED jfsh_binaries_download_total 1698850564126\n# TYPE jfsh_binaries_download_total counter\njfsh_binaries_download_total{id="s3-storage-v3",name="s3_storage_v3"} 2583 1698850569605\n# HELP jfsh_binaries_upload_total Counts binaries uploads\n# UPDATED jfsh_binaries_upload_total 1698850564126\n# TYPE jfsh_binaries_upload_total counter\njfsh_binaries_upload_total{id="s3-storage-v3",name="s3_storage_v3"} 2066 1698850569605\n# HELP jfsh_signed_url_redirect_total Counts how many times signed url was generated\n# UPDATED jfsh_signed_url_redirect_total 1698850564126\n# TYPE jfsh_signed_url_redirect_total counter\njfsh_signed_url_redirect_total{id="s3-storage-v3",name="s3_storage_v3"} 0 1698850569605\n# HELP jfrt_db_connections_active_total Total Active Connections\n# UPDATED jfrt_db_connections_active_total 1698850564126\n# TYPE jfrt_db_connections_active_total gauge\njfrt_db_connections_active_total 0 1698850569605\n# HELP jfrt_db_connections_idle_total Total Idle Connections\n# UPDATED jfrt_db_connections_idle_total 1698850564126\n# TYPE jfrt_db_connections_idle_total gauge\njfrt_db_connections_idle_total 2 1698850569605\n# HELP jfrt_db_connections_max_active_total Total Max Active Connections\n# UPDATED jfrt_db_connections_max_active_total 1698850564126\n# TYPE jfrt_db_connections_max_active_total gauge\njfrt_db_connections_max_active_total 300 1698850569605\n# HELP jfrt_db_connections_min_idle_total Total Min Idle Connections\n# UPDATED jfrt_db_connections_min_idle_total 1698850564126\n# TYPE jfrt_db_connections_min_idle_total gauge\njfrt_db_connections_min_idle_total 1 1698850569605\n# HELP app_disk_used_bytes Used bytes for app home directory disk device\n# UPDATED app_disk_used_bytes 1698850569605\n# TYPE app_disk_used_bytes gauge\napp_disk_used_bytes 79492304896 1698850569605\n# HELP app_disk_free_bytes Free bytes for app home directory disk device\n# UPDATED app_disk_free_bytes 1698850569605\n# TYPE app_disk_free_bytes gauge\napp_disk_free_bytes 130749255680 1698850569605\n# HELP adoption_artifactory_authentication_count Authentication\n# UPDATED adoption_artifactory_authentication_count 1698850569605\n# TYPE adoption_artifactory_authentication_count counter\nadoption_artifactory_authentication_count{action="login",apiKey="false",domain="adoption",realm="internal",token="true"} 33 1698850569605\n# HELP adoption_artifactory_authentication_count Authentication\n# UPDATED adoption_artifactory_authentication_count 1698850569605\n# TYPE adoption_artifactory_authentication_count counter\nadoption_artifactory_authentication_count{action="login",apiKey="true",domain="adoption",realm="oauth",token="false"} 1 1698850569605\n# HELP adoption_artifactory_authentication_count Authentication\n# UPDATED adoption_artifactory_authentication_count 1698850569605\n# TYPE adoption_artifactory_authentication_count counter\nadoption_artifactory_authentication_count{action="login",apiKey="false",domain="adoption",realm="oauth",token="true"} 8779 1698850569605\n# HELP jfsh_thread_pool_thread_pool_active_count_total Number of threads that are actively executing tasks\n# UPDATED jfsh_thread_pool_thread_pool_active_count_total 1698850564126\n# TYPE jfsh_thread_pool_thread_pool_active_count_total gauge\njfsh_thread_pool_thread_pool_active_count_total{maxPoolSize="3.0",mergedPools="0",name="binary-store-gc"} 0.0 1698850569605\n# HELP jfsh_thread_pool_thread_pool_pending_tasks_total Number of tasks pending execution\n# UPDATED jfsh_thread_pool_thread_pool_pending_tasks_total 1698850564126\n# TYPE jfsh_thread_pool_thread_pool_pending_tasks_total gauge\njfsh_thread_pool_thread_pool_pending_tasks_total{maxPoolSize="3.0",mergedPools="0",name="binary-store-gc"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_created_seconds RB V2 version creation time: percentile 50%\n# UPDATED jfbh_rbv2_version_created_seconds 1698850564127\n# TYPE jfbh_rbv2_version_created_seconds gauge\njfbh_rbv2_version_created_seconds{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_created_seconds RB V2 version creation time: percentile 75%\n# UPDATED jfbh_rbv2_version_created_seconds 1698850564127\n# TYPE jfbh_rbv2_version_created_seconds gauge\njfbh_rbv2_version_created_seconds{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_created_seconds RB V2 version creation time: percentile 95%\n# UPDATED jfbh_rbv2_version_created_seconds 1698850564127\n# TYPE jfbh_rbv2_version_created_seconds gauge\njfbh_rbv2_version_created_seconds{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_created_seconds RB V2 version creation time: percentile 99%\n# UPDATED jfbh_rbv2_version_created_seconds 1698850564127\n# TYPE jfbh_rbv2_version_created_seconds gauge\njfbh_rbv2_version_created_seconds{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_created_seconds RB V2 version creation time: standard deviation\n# UPDATED jfbh_rbv2_version_created_seconds 1698850564127\n# TYPE jfbh_rbv2_version_created_seconds gauge\njfbh_rbv2_version_created_seconds{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_created_seconds RB V2 promotion creation time: percentile 50%\n# UPDATED jfbh_rbv2_promotion_created_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_created_seconds gauge\njfbh_rbv2_promotion_created_seconds{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_created_seconds RB V2 promotion creation time: percentile 75%\n# UPDATED jfbh_rbv2_promotion_created_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_created_seconds gauge\njfbh_rbv2_promotion_created_seconds{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_created_seconds RB V2 promotion creation time: percentile 95%\n# UPDATED jfbh_rbv2_promotion_created_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_created_seconds gauge\njfbh_rbv2_promotion_created_seconds{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_created_seconds RB V2 promotion creation time: percentile 99%\n# UPDATED jfbh_rbv2_promotion_created_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_created_seconds gauge\njfbh_rbv2_promotion_created_seconds{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_created_seconds RB V2 promotion creation time: standard deviation\n# UPDATED jfbh_rbv2_promotion_created_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_created_seconds gauge\njfbh_rbv2_promotion_created_seconds{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_deleted_seconds RB V2 promotion deletion time: percentile 50%\n# UPDATED jfbh_rbv2_promotion_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_deleted_seconds gauge\njfbh_rbv2_promotion_deleted_seconds{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_deleted_seconds RB V2 promotion deletion time: percentile 75%\n# UPDATED jfbh_rbv2_promotion_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_deleted_seconds gauge\njfbh_rbv2_promotion_deleted_seconds{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_deleted_seconds RB V2 promotion deletion time: percentile 95%\n# UPDATED jfbh_rbv2_promotion_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_deleted_seconds gauge\njfbh_rbv2_promotion_deleted_seconds{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_deleted_seconds RB V2 promotion deletion time: percentile 99%\n# UPDATED jfbh_rbv2_promotion_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_deleted_seconds gauge\njfbh_rbv2_promotion_deleted_seconds{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_promotion_deleted_seconds RB V2 promotion deletion time: standard deviation\n# UPDATED jfbh_rbv2_promotion_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_promotion_deleted_seconds gauge\njfbh_rbv2_promotion_deleted_seconds{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_deleted_seconds RB V2 version deletion time: percentile 50%\n# UPDATED jfbh_rbv2_version_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_version_deleted_seconds gauge\njfbh_rbv2_version_deleted_seconds{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_deleted_seconds RB V2 version deletion time: percentile 75%\n# UPDATED jfbh_rbv2_version_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_version_deleted_seconds gauge\njfbh_rbv2_version_deleted_seconds{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_deleted_seconds RB V2 version deletion time: percentile 95%\n# UPDATED jfbh_rbv2_version_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_version_deleted_seconds gauge\njfbh_rbv2_version_deleted_seconds{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_deleted_seconds RB V2 version deletion time: percentile 99%\n# UPDATED jfbh_rbv2_version_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_version_deleted_seconds gauge\njfbh_rbv2_version_deleted_seconds{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_version_deleted_seconds RB V2 version deletion time: standard deviation\n# UPDATED jfbh_rbv2_version_deleted_seconds 1698850564127\n# TYPE jfbh_rbv2_version_deleted_seconds gauge\njfbh_rbv2_version_deleted_seconds{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifact_copied_seconds RB V2 artifact copy time: percentile 50%\n# UPDATED jfbh_rbv2_artifact_copied_seconds 1698850564127\n# TYPE jfbh_rbv2_artifact_copied_seconds gauge\njfbh_rbv2_artifact_copied_seconds{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifact_copied_seconds RB V2 artifact copy time: percentile 75%\n# UPDATED jfbh_rbv2_artifact_copied_seconds 1698850564127\n# TYPE jfbh_rbv2_artifact_copied_seconds gauge\njfbh_rbv2_artifact_copied_seconds{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifact_copied_seconds RB V2 artifact copy time: percentile 95%\n# UPDATED jfbh_rbv2_artifact_copied_seconds 1698850564127\n# TYPE jfbh_rbv2_artifact_copied_seconds gauge\njfbh_rbv2_artifact_copied_seconds{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifact_copied_seconds RB V2 artifact copy time: percentile 99%\n# UPDATED jfbh_rbv2_artifact_copied_seconds 1698850564127\n# TYPE jfbh_rbv2_artifact_copied_seconds gauge\njfbh_rbv2_artifact_copied_seconds{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifact_copied_seconds RB V2 artifact copy time: standard deviation\n# UPDATED jfbh_rbv2_artifact_copied_seconds 1698850564127\n# TYPE jfbh_rbv2_artifact_copied_seconds gauge\njfbh_rbv2_artifact_copied_seconds{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifacts_total RB V2 artifacts: percentile 50%\n# UPDATED jfbh_rbv2_artifacts_total 1698850564127\n# TYPE jfbh_rbv2_artifacts_total gauge\njfbh_rbv2_artifacts_total{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifacts_total RB V2 artifacts: percentile 75%\n# UPDATED jfbh_rbv2_artifacts_total 1698850564127\n# TYPE jfbh_rbv2_artifacts_total gauge\njfbh_rbv2_artifacts_total{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifacts_total RB V2 artifacts: percentile 95%\n# UPDATED jfbh_rbv2_artifacts_total 1698850564127\n# TYPE jfbh_rbv2_artifacts_total gauge\njfbh_rbv2_artifacts_total{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifacts_total RB V2 artifacts: percentile 99%\n# UPDATED jfbh_rbv2_artifacts_total 1698850564127\n# TYPE jfbh_rbv2_artifacts_total gauge\njfbh_rbv2_artifacts_total{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_artifacts_total RB V2 artifacts: standard deviation\n# UPDATED jfbh_rbv2_artifacts_total 1698850564127\n# TYPE jfbh_rbv2_artifacts_total gauge\njfbh_rbv2_artifacts_total{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_properties_total RB V2 properties: percentile 50%\n# UPDATED jfbh_rbv2_properties_total 1698850564127\n# TYPE jfbh_rbv2_properties_total gauge\njfbh_rbv2_properties_total{quantile="0.5"} 0.0 1698850569605\n# HELP jfbh_rbv2_properties_total RB V2 properties: percentile 75%\n# UPDATED jfbh_rbv2_properties_total 1698850564127\n# TYPE jfbh_rbv2_properties_total gauge\njfbh_rbv2_properties_total{quantile="0.75"} 0.0 1698850569605\n# HELP jfbh_rbv2_properties_total RB V2 properties: percentile 95%\n# UPDATED jfbh_rbv2_properties_total 1698850564127\n# TYPE jfbh_rbv2_properties_total gauge\njfbh_rbv2_properties_total{quantile="0.95"} 0.0 1698850569605\n# HELP jfbh_rbv2_properties_total RB V2 properties: percentile 99%\n# UPDATED jfbh_rbv2_properties_total 1698850564127\n# TYPE jfbh_rbv2_properties_total gauge\njfbh_rbv2_properties_total{quantile="0.99"} 0.0 1698850569605\n# HELP jfbh_rbv2_properties_total RB V2 properties: standard deviation\n# UPDATED jfbh_rbv2_properties_total 1698850564127\n# TYPE jfbh_rbv2_properties_total gauge\njfbh_rbv2_properties_total{type="std"} 0.0 1698850569605\n# HELP jfbh_rbv2_creation_methods_total RB V2 creation methods\n# UPDATED jfbh_rbv2_creation_methods_total 1698850564127\n# TYPE jfbh_rbv2_creation_methods_total counter\njfbh_rbv2_creation_methods_total{source_method="artifacts"} 0 1698850569605\n# HELP jfbh_rbv2_creation_methods_total RB V2 creation methods\n# UPDATED jfbh_rbv2_creation_methods_total 1698850564127\n# TYPE jfbh_rbv2_creation_methods_total counter\njfbh_rbv2_creation_methods_total{source_method="release_bundles"} 0 1698850569605\n# HELP jfbh_rbv2_creation_methods_total RB V2 creation methods\n# UPDATED jfbh_rbv2_creation_methods_total 1698850564127\n# TYPE jfbh_rbv2_creation_methods_total counter\njfbh_rbv2_creation_methods_total{source_method="aql"} 0 1698850569605\n# HELP jfbh_rbv2_creation_methods_total RB V2 creation methods\n# UPDATED jfbh_rbv2_creation_methods_total 1698850564127\n# TYPE jfbh_rbv2_creation_methods_total counter\njfbh_rbv2_creation_methods_total{source_method="builds"} 0 1698850569605\n# HELP jfbh_rbv2_creation_agents_total RB V2 creation agents\n# UPDATED jfbh_rbv2_creation_agents_total 1698850564127\n# TYPE jfbh_rbv2_creation_agents_total counter\njfbh_rbv2_creation_agents_total{source_agent="api"} 0 1698850569605\n# HELP jfbh_rbv2_creation_agents_total RB V2 creation agents\n# UPDATED jfbh_rbv2_creation_agents_total 1698850564127\n# TYPE jfbh_rbv2_creation_agents_total counter\njfbh_rbv2_creation_agents_total{source_agent="browser"} 0 1698850569605\n# HELP jfbh_rbv2_creation_agents_total RB V2 creation agents\n# UPDATED jfbh_rbv2_creation_agents_total 1698850564127\n# TYPE jfbh_rbv2_creation_agents_total counter\njfbh_rbv2_creation_agents_total{source_agent="cli"} 0 1698850569605\n# HELP jfbh_rbv2_bundle_operations_total RB V2 bundle operations\n# UPDATED jfbh_rbv2_bundle_operations_total 1698850564127\n# TYPE jfbh_rbv2_bundle_operations_total counter\njfbh_rbv2_bundle_operations_total{operation="create_bundle"} 0 1698850569605\n# HELP jfbh_rbv2_bundle_operations_total RB V2 bundle operations\n# UPDATED jfbh_rbv2_bundle_operations_total 1698850564127\n# TYPE jfbh_rbv2_bundle_operations_total counter\njfbh_rbv2_bundle_operations_total{operation="delete_bundle"} 0 1698850569605\n# HELP jfbh_rbv2_promotion_operations_total RB V2 promotion operations\n# UPDATED jfbh_rbv2_promotion_operations_total 1698850564127\n# TYPE jfbh_rbv2_promotion_operations_total counter\njfbh_rbv2_promotion_operations_total{operation="delete_promotion"} 0 1698850569605\n# HELP jfbh_rbv2_promotion_operations_total RB V2 promotion operations\n# UPDATED jfbh_rbv2_promotion_operations_total 1698850564127\n# TYPE jfbh_rbv2_promotion_operations_total counter\njfbh_rbv2_promotion_operations_total{operation="create_promotion"} 0 1698850569605\n# HELP jfbh_rbv2_release_bundles_total RB V2 release bundles\n# UPDATED jfbh_rbv2_release_bundles_total 1698850564127\n# TYPE jfbh_rbv2_release_bundles_total gauge\njfbh_rbv2_release_bundles_total 0 1698850569605\n# HELP jfbh_rbv2_release_bundles_completed_total RB V2 release bundles completed\n# UPDATED jfbh_rbv2_release_bundles_completed_total 1698850564127\n# TYPE jfbh_rbv2_release_bundles_completed_total gauge\njfbh_rbv2_release_bundles_completed_total 0 1698850569605\n# HELP jfbh_rbv2_release_bundles_failed_total RB V2 release bundles failed\n# UPDATED jfbh_rbv2_release_bundles_failed_total 1698850564127\n# TYPE jfbh_rbv2_release_bundles_failed_total gauge\njfbh_rbv2_release_bundles_failed_total 0 1698850569605\n# HELP jfbh_rbv2_promotions_total RB V2 promotions\n# UPDATED jfbh_rbv2_promotions_total 1698850564127\n# TYPE jfbh_rbv2_promotions_total gauge\njfbh_rbv2_promotions_total 0 1698850569605\n# HELP jfbh_rbv2_promotions_completed_total RB V2 promotions completed\n# UPDATED jfbh_rbv2_promotions_completed_total 1698850564127\n# TYPE jfbh_rbv2_promotions_completed_total gauge\njfbh_rbv2_promotions_completed_total 0 1698850569605\n# HELP jfbh_rbv2_promotions_failed_total RB V2 promotions failed\n# UPDATED jfbh_rbv2_promotions_failed_total 1698850564127\n# TYPE jfbh_rbv2_promotions_failed_total gauge\njfbh_rbv2_promotions_failed_total 0 1698850569605\n# HELP jfrt_events_gc_size_cleaned_bytes Total Bytes recovered by Garbage Collection\n# UPDATED jfrt_events_gc_size_cleaned_bytes 1698850564126\n# TYPE jfrt_events_gc_size_cleaned_bytes gauge\njfrt_events_gc_size_cleaned_bytes{type="EVENTS_GC"} 0 1698850569605\n# HELP jfrt_events_gc_binaries_deleted_total Total number of binaries removed by Garbage Collection\n# UPDATED jfrt_events_gc_binaries_deleted_total 1698850564126\n# TYPE jfrt_events_gc_binaries_deleted_total counter\njfrt_events_gc_binaries_deleted_total{type="EVENTS_GC"} 0 1698850569605\n# HELP jfrt_events_gc_waiting_total Total number of waiting event\n# UPDATED jfrt_events_gc_waiting_total 1698850564126\n# TYPE jfrt_events_gc_waiting_total gauge\njfrt_events_gc_waiting_total{type="EVENTS_GC"} 0 1698850569605\n# HELP jfrt_events_gc_events_total Total number of event\n# UPDATED jfrt_events_gc_events_total 1698850564126\n# TYPE jfrt_events_gc_events_total counter\njfrt_events_gc_events_total{type="EVENTS_GC"} 0 1698850569605\n# HELP jfrt_runtime_heap_freememory_bytes Free Memory\n# UPDATED jfrt_runtime_heap_freememory_bytes 1698850569605\n# TYPE jfrt_runtime_heap_freememory_bytes gauge\njfrt_runtime_heap_freememory_bytes 3532722600 1698850569605\n# HELP jfrt_runtime_heap_maxmemory_bytes Max Memory\n# UPDATED jfrt_runtime_heap_maxmemory_bytes 1698850569605\n# TYPE jfrt_runtime_heap_maxmemory_bytes gauge\njfrt_runtime_heap_maxmemory_bytes 12884901888 1698850569605\n# HELP jfrt_runtime_heap_totalmemory_bytes Total Memory\n# UPDATED jfrt_runtime_heap_totalmemory_bytes 1698850569605\n# TYPE jfrt_runtime_heap_totalmemory_bytes gauge\njfrt_runtime_heap_totalmemory_bytes 8589934592 1698850569605\n# HELP jfrt_runtime_heap_processors_total Available Processors\n# UPDATED jfrt_runtime_heap_processors_total 1698850569605\n# TYPE jfrt_runtime_heap_processors_total counter\njfrt_runtime_heap_processors_total 4 1698850569605\n# HELP jfrt_projects_active_total Projects Amount\n# UPDATED jfrt_projects_active_total 1698850569605\n# TYPE jfrt_projects_active_total counter\njfrt_projects_active_total 1 1698850569605\n# HELP jfrt_storage_current_total_size_bytes Used Storage\n# UPDATED jfrt_storage_current_total_size_bytes 1698850569605\n# TYPE jfrt_storage_current_total_size_bytes gauge\njfrt_storage_current_total_size_bytes 0 1698850569605\n# HELP jfrt_artifacts_gc_next_run_seconds Next GC Run\n# UPDATED jfrt_artifacts_gc_next_run_seconds 1698840003794\n# TYPE jfrt_artifacts_gc_next_run_seconds gauge\njfrt_artifacts_gc_next_run_seconds 14396 1698850569605\n# HELP jfrt_artifacts_gc_duration_seconds Time taken for garbage collection\n# UPDATED jfrt_artifacts_gc_duration_seconds 1698840003794\n# TYPE jfrt_artifacts_gc_duration_seconds gauge\njfrt_artifacts_gc_duration_seconds{end="1698840000346",start="1698840000005",status="COMPLETED",type="TRASH_AND_BINARIES"} 0.341 1698850569605\n# HELP jfrt_artifacts_gc_size_cleaned_bytes Total Bytes recovered by Garbage Collection\n# UPDATED jfrt_artifacts_gc_size_cleaned_bytes 1698840003794\n# TYPE jfrt_artifacts_gc_size_cleaned_bytes gauge\njfrt_artifacts_gc_size_cleaned_bytes{end="1698840000346",start="1698840000005",status="COMPLETED",type="TRASH_AND_BINARIES"} 0 1698850569605\n# HELP jfrt_artifacts_gc_binaries_total Total number of binaries removed by Garbage Collection\n# UPDATED jfrt_artifacts_gc_binaries_total 1698840003794\n# TYPE jfrt_artifacts_gc_binaries_total gauge\njfrt_artifacts_gc_binaries_total{end="1698840000346",start="1698840000005",status="COMPLETED",type="TRASH_AND_BINARIES"} 0 1698850569605\n"

CURL OF /METRICS:
jcocozzi@jcocozzi ansible % curl http://localhost:33333/metrics
# HELP artifactory_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which artifactory_exporter was built.
# TYPE artifactory_exporter_build_info gauge
artifactory_exporter_build_info{branch="refs/heads/master",goversion="go1.18.10",revision="90e6a9c9c3499ef972ad5da472faf4dd6924e26d",version="canary"} 1
# HELP artifactory_exporter_json_parse_failures Number of errors while parsing Json.
# TYPE artifactory_exporter_json_parse_failures counter
artifactory_exporter_json_parse_failures 0
# HELP artifactory_exporter_total_api_errors Current total API errors.
# TYPE artifactory_exporter_total_api_errors counter
artifactory_exporter_total_api_errors 0
# HELP artifactory_exporter_total_scrapes Current total artifactory scrapes.
# TYPE artifactory_exporter_total_scrapes counter
artifactory_exporter_total_scrapes 88
# HELP artifactory_security_groups Number of Artifactory groups
# TYPE artifactory_security_groups gauge
artifactory_security_groups{node_id="artifactory-primary-0"} 4
# HELP artifactory_security_users Number of Artifactory users for each realm.
# TYPE artifactory_security_users gauge
artifactory_security_users{node_id="artifactory-primary-0",realm="internal"} 4
artifactory_security_users{node_id="artifactory-primary-0",realm="oauth"} 21
# HELP artifactory_up Was the last scrape of artifactory successful.
# TYPE artifactory_up gauge
artifactory_up 0
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 8.0219e-05
go_gc_duration_seconds{quantile="0.25"} 0.000115655
go_gc_duration_seconds{quantile="0.5"} 0.000136795
go_gc_duration_seconds{quantile="0.75"} 0.000160449
go_gc_duration_seconds{quantile="1"} 0.000309164
go_gc_duration_seconds_sum 0.009332442
go_gc_duration_seconds_count 65
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 10
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.18.10"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.742808e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.0376784e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 4264
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 226127
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 3.4429351732669383e-06
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.001792e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.742808e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 7.397376e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.202496e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 2975
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.85728e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.1599872e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.698855069602287e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 229102
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 28800
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 31200
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 207648
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 227808
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.66616e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 2.028648e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 983040
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 983040
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.9876624e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 18
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1.98
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 11
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.1178752e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.69884999035e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.94148352e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 87
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
@jcocozzi
Copy link
Author

jcocozzi commented Nov 1, 2023

Tagging @davidshadix as it looks like you created the PR and may have a suggestion.

@davidshadix
Copy link
Contributor

Hmm not sure. I will try to take a look soon. In the meantime, can you share your config yaml file?

@davidshadix
Copy link
Contributor

Looks like the script is failing to parse the jfrog open metrics now. I'm not 100% but I think the open metrics format changed but not sure what downstream issues that has caused. I may need to revaluate how I'm parsing the metrics.

@jcocozzi
Copy link
Author

jcocozzi commented Nov 6, 2023

Thanks for taking a look. Sounds like you don't need my config yaml any longer, but if you do, let me know.

@JyotsnaSharma21
Copy link

I am facing issue related to this. I installed it using helm charts. My values.yaml has 'open_metrics' like this-
options:
logLevel: info
logFormat: logfmt
telemetryPath: /metrics
verifySSL: true
timeout: 10s
optionalMetrics:
- artifacts
- open_metrics
This gives error in pod's logs-
level=error msg="Error creating the config. err: unknown optional metric: open_metrics. Valid optional metrics are: [artifacts replication_status federation_status]" source="artifactory_exporter.go:20"

@jcocozzi
Copy link
Author

jcocozzi commented Nov 9, 2023

@JyotsnaSharma21 - This sounds like you are not running with the canary version of the exporter. If you are deploying using the helm chart you will want to set the tag to canary rather than v1.13.2.

image:
registry: ghcr.io
repository: peimanja/artifactory_exporter
# set to canary for the latest unreleased version
tag: v1.13.2 <------------------------------------------------
pullPolicy: IfNotPresent

The open_metrics functionality is not yet in a release.

@JyotsnaSharma21
Copy link

Thanks @jcocozzi
It is deployed successfully now. But I can't find any new metrics on Prometheus with this change.
Could you suggest, what are the expected metrics with this change?

Copy link

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions bot added the Stale label Dec 11, 2023
@jcocozzi
Copy link
Author

@JyotsnaSharma21 - This is the reason I opened the ticket. The new metrics are not functioning.

@davidshadix - Any chance you are going to find the time to fix this? If not, I will try and find some time.

@github-actions github-actions bot removed the Stale label Dec 16, 2023
Copy link

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions bot added the Stale label Jan 15, 2024
@peimanja peimanja removed the Stale label Jan 15, 2024
Copy link

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions bot added the Stale label Feb 15, 2024
@peimanja peimanja removed the Stale label Feb 16, 2024
@peimanja
Copy link
Owner

I just gave open_metrics a try and looks like it's showing up without any issue. @jcocozzi what version/license of Artifactory are you running ?

also can you check all the logs for errors? judging from artifactory_up 0 looks like it failed at some point to scrape all the metrics.

# HELP artifactory_up Was the last scrape of artifactory successful.
# TYPE artifactory_up gauge
artifactory_up 0

@ltang-wise
Copy link

ltang-wise commented Mar 15, 2024

Adding to this thread as I'm having the exact same issue as this one - @jcocozzi did you find any solution to this?

The service logs logs shows the metrics being scraped but the are unavailable to our prometheus instance. I'm using exporter v1.14.0.

systemctl status docker.arti_repo_level_exporter.service
... 
Mar 14 21:03:46 ip-***-**-**-*** docker[*****]: time=2024-03-14T21:03:46.166Z level=DEBUG msg="OpenMetrics from Artifactory util" body="# HELP jfsh_binaries_download_total Counts the total binaries download\n# UPDATED jfsh_binaries_download_total 1710450200285\n# TYPE jfsh_binaries_download_total counter\njfsh_b>

However, when I curl the ≤scrape_port>/metrics endpoint These don't appear:

curl http://localhost:9531/metrics
# HELP artifactory_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which artifactory_exporter was built.
# TYPE artifactory_exporter_build_info gauge
artifactory_exporter_build_info{branch="refs/tags/v1.14.0",goversion="go1.21.7",revision="8dfb2cae07db3d6c03244ea8b2d450b39c716dbe",version="v1.14.0"} 1
# HELP artifactory_exporter_json_parse_failures Number of errors while parsing Json.
# TYPE artifactory_exporter_json_parse_failures counter
artifactory_exporter_json_parse_failures 0
# HELP artifactory_exporter_total_api_errors Current total API errors.
# TYPE artifactory_exporter_total_api_errors counter
artifactory_exporter_total_api_errors 0
# HELP artifactory_exporter_total_scrapes Current total artifactory scrapes.
# TYPE artifactory_exporter_total_scrapes counter
artifactory_exporter_total_scrapes 79
# HELP artifactory_replication_enabled Replication status for an Artifactory repository (1 = enabled).
# TYPE artifactory_replication_enabled gauge
... etc

These do come up in the docker logs container logs too:

docker logs arti-repo-level-exporter 
time=2024-03-15T14:26:12.326Z level=DEBUG msg="OpenMetrics from Artifactory" body="# HELP jfsh_binaries_download_total Counts the total binaries download\n# UPDATED jfsh_binaries_download_total 1710512767694\n# TYPE jfsh_binaries_download_total counter\njfsh_binaries_download_total{id=\"cache-fs\",name=\"cache-fs\"} 2005582 1710512772325\n# HELP jfsh_binaries_not_found_total Counts the total request not found\n# UPDATED jfsh_binaries_not_found_total 1710512767694\n# TYPE jfsh_binaries_not_found_total counter\njfsh_binaries_not_found_total{id=\"cache-fs\",name=\"cache-fs\"} 150722 1710512772325\n# HELP jfsh_cache_evicted_bytes Counts the amount of cleared bytes from cache-fs\n# UPDATED jfsh_cache_evicted_bytes 1710512767694\n# TYPE jfsh_cache_evicted_bytes gauge\njfsh_cache_evicted_bytes{id=\"cache-fs\",name=\"cache-fs\"} 364292261406 1710512772325\n# HELP jfsh_storage_size_bytes Size (bytes) in storage\n# UPDATED jfsh_storage_size_bytes 1710512767694\n# TYPE jfsh_storage_size_bytes gauge\njfsh_storage_size_bytes{id=\"cache-fs\",name=\"cache-fs\"} 99903984457 1710512772325\n# HELP jfsh_binaries_count_total Number of binaries in storage\n# UPDATED jfsh_binaries_count_total 1710512767694\n# TYPE jfsh_binaries_count_total gauge\njfsh_binaries_count_total{id=\"cache-fs\",name=\"cache-fs\"} 34251 1710512772325\n
... etc

Our setup is slightly different though - We don't use k8s - we run Artifactory EnterpriseX v7.71.11 self-hosted in a highly available configuration on 3 AWS EC2 nodes. We pull the upstream docker image directly from a proxied remote artifactory repo and specify the environment variables in a service file, which all lives in an ansible repo. This gets baked into an AWS AMI which we run on the EC2 nodes.

Our service file looks like this:

[Unit]
Description=Arti repo exporter service in a container
After=docker.service docker.artifactory.service
Requires=docker.service

[Service]
TimeoutStartSec=10
User=artifactory
Group=artifactory
Restart=always
ExecStartPre=-/usr/bin/docker stop arti-repo-level-exporter
ExecStartPre=-/usr/bin/docker rm arti-repo-level-exporter
ExecStart=/usr/bin/docker run --rm --name arti-repo-level-exporter \
    -e artifactory.ssl-verify=false \
    -e ARTI_USERNAME="prometheus" \
    -e ARTI_PASSWORD=PROMETHEUS_USER_PASSWD \
    ## Port below is not in use since network is host mode
    # -p 9531:9531 \
    --network=host \
    {{ arti_repo_level_exporter_image }}:{{ arti_repo_level_exporter_version }} \
    --log.level=debug \
    --optional-metric=open_metrics
ExecStop=/usr/bin/docker stop arti-repo-level-exporter

[Install]
WantedBy=multi-user.target

@ltang-wise
Copy link

Hey @peimanja is there chance you have any suspicion what could be causing this issue? I've noticed that we have this log line when curling the exporter metrics endpoint:

curl http://localhost:9531/metrics
...
# HELP artifactory_up Was the last scrape of artifactory successful.
# TYPE artifactory_up gauge
artifactory_up 0

@peimanja
Copy link
Owner

@ltang-wise that just shows there was an issue with the scrape. you nee to check the logs on the exporter. I suggest you enable the debug log on the exporter and run the curl again and see the logs of the exporter.

@kkaresz-tw
Copy link

Hi @peimanja ! After some debugging with @ltang-wise I thought I'd summarise our findings.

The reason we didn't see anything in the logs was that the scraper silently fails in case of an error - in our case here: https://github.com/peimanja/artifactory_exporter/blob/master/collector/collector.go#L173-L178. I wrote "silently" because the collector does not log when it hits an error here: https://github.com/peimanja/artifactory_exporter/blob/master/collector/openMetrics.go#L28-L30. So our symptom was that we saw that the openMetrics string was logged (https://github.com/peimanja/artifactory_exporter/blob/master/collector/openMetrics.go#L19-L22) but nothing happened after that. Adding some more debug logs in the error clause revealed an error which made us realise that the collectors after optionalMetrics (e.g. the system metrics https://github.com/peimanja/artifactory_exporter/blob/master/collector/collector.go#L181) did not even get scraped.

The error we got was:

msg="!BADKEY="text format parsing error in line 21: second HELP line for metric name \"jfsh_binaries_download_total\""

which made us also realise that we hit the same issue that we've hit before with our own in-house custom Artifactory exporter we were about to replace with this one.

The issue we hit is basically that the Artifactory openMetrics endpoint returns metrics with the same name, but with different labels, and Prometheus does not like this for some reason. (Maybe the TextToMetricFamilies function is not implemented properly?) The above error mentions that the HELP line is duplicated, but I guess the main problem is that the actual gauge is duplicated, and we don't exactly know what to do with this. The body of openMetrics.PromMetrics has these (and a few other) metrics that are duplicated:

...
# HELP jfsh_binaries_download_total Counts the total binaries download
# UPDATED jfsh_binaries_download_total 1711450800961
# TYPE jfsh_binaries_download_total counter
jfsh_binaries_download_total{id=\"cache-fs\",name=\"cache-fs\"} 23022116 1711450837097

...
...
...

# HELP jfsh_binaries_download_total Counts the total binaries download
# UPDATED jfsh_binaries_download_total 1711450800961
# TYPE jfsh_binaries_download_total counter
jfsh_binaries_download_total{id=\"s3-storage-v3\",name=\"s3-storage-v3\"} 1535024 1711450837097
...

At this point we're struggling to figure out whether it's our Artifactory configuaration that is bad, or is it Artifactory's openMetrics endpoint/response that is not okay, or if we're not handling Prometheus gauges properly.

If you have any insights about this problem, any input would be appreciated. At this point the only "fix" within this exporter I could contribute with is just a few more lines of debug logs. I'm not sure what to do with the duplication.

If there is an open issue about this already that I missed, please let me know and I'll link it properly. Thank you!

Copy link

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions bot added the Stale label Apr 26, 2024
@peimanja peimanja removed the Stale label Apr 26, 2024
Copy link

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions bot added the Stale label May 27, 2024
@Momotoculteur
Copy link

Hey !
Got also this issue, open_metrics is not scrapped in my /metrics path :(

@github-actions github-actions bot removed the Stale label May 28, 2024
@peimanja peimanja added the keep-open Make stale action ignore the Issue/PR label May 28, 2024
@jcocozzi
Copy link
Author

Unfortunately I have had the time to revisit this with the latest version of Artifactory and latest version of the exporter to see if it is indeed still an issue for me. Hopefully I get a chance to revisit this soon. Apologies for not following up.

@antonchev
Copy link

Hello! confirm that the problem is relevant on the version artifactory:v7.84.14 and exporter:v1.14.0
help solve the problem, thx

@ex12is
Copy link

ex12is commented Aug 22, 2024

Same problem - artifactory:v7.90.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
keep-open Make stale action ignore the Issue/PR Need Investigation
Projects
None yet
Development

No branches or pull requests

9 participants