Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can this be docked with Grafana？ #4

Open
nanquanyuhao opened this issue Apr 24, 2018 · 20 comments
Open

Can this be docked with Grafana？ #4

nanquanyuhao opened this issue Apr 24, 2018 · 20 comments

Comments

@nanquanyuhao
Copy link

nanquanyuhao commented Apr 24, 2018 •
edited
Loading

Hello...

With your component radosgw_usage_exporter , I have pull data with Prometheus.
But I wanna view data with "grafana + Prometheus", do you have any dashboard mix this exporter? Can you help me ?

Thanks!

tim 20180424112914
By the way , why do I only see these data?
@blemmenes
Copy link
Owner

Hello,

You can certainly use Grafana to view these metrics, I have not however built any grafana dashboards specific to the metrics produced by this exporter. It is on my list of things to do, but not something I will get to in the next week or two.

Are those the extent of the metrics you get back? Can you provide info on what ceph version this is running against?

Also, does the user you are using with this module have both usage= read and buckets=read capabilities?

Thanks,
Berant

@nanquanyuhao
Copy link
Author

@blemmenes Thanks for your response...

That picture content is indeed all the data... And I use the docker image ceph/demo:tag-build-stable-2.2.11-jewel-centos-7 from dockehub. The account I used info like this and I think it has usage= read and buckets=read capabilities:

{
    "user_id": "admin",
    "display_name": "Administrator",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "admin",
            "access_key": "6H23GNHCC1QTMSRJUSHX",
            "secret_key": "arKl2acDMKnrUsBDQGJVxO6U28m8txXzxEdw32lN"
        }
    ],
    "swift_keys": [],
    "caps": [
        {
            "type": "buckets",
            "perm": "*"
        },
        {
            "type": "metadata",
            "perm": "*"
        },
        {
            "type": "usage",
            "perm": "*"
        },
        {
            "type": "users",
            "perm": "*"
        },
        {
            "type": "zone",
            "perm": "*"
        }
    ],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

But I hadn't add rgw enable usage log = true to ceph.conf, is that the reason?

Next I will install your component radosgw_usage_exporter directly on my ceph gluster in VM...

By the way , My English is very poor, Thanks for your help again...

@blemmenes
Copy link
Owner

Yeah, if you haven't enabled the usage log you won't get any of that data. Let me know if you have any issues once you enable that.

No worries on your English.

Thanks,
Berant

@nanquanyuhao
Copy link
Author

@blemmenes
But as you saw, I have some data like this only this... Is it all?

# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 179810304.0
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 15400960.0
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1524539063.39
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 101.48
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 7.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 65536.0
# HELP radosgw_usage_successful_ops_total Number of successful operations
# TYPE radosgw_usage_successful_ops_total counter
# HELP radosgw_usage_bucket_total Bucket used bytes
# TYPE radosgw_usage_bucket_total gauge
radosgw_usage_bucket_total{bucket="test10",owner="1_1_1524067231262",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test2",owner="1_1_1523696628085",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="my-new-bucket",owner="testuser",zonegroup="0"} 40960.0
radosgw_usage_bucket_total{bucket="test7",owner="1_1_1524066529235",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test11",owner="1_1_1524067371106",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test4",owner="1_1_1523697318438",zonegroup="0"} 36864.0
radosgw_usage_bucket_total{bucket="tset12",owner="1_1_1524067397231",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="lis001",owner="1_1_1524651275973",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test9",owner="1_1_1524067064272",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test",owner="1_1_1523692816592",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="myfirstbucket",owner="1_1_1523619384379",zonegroup="0"} 40960.0
radosgw_usage_bucket_total{bucket="thsd11",owner="3_3_1524362118250",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test8",owner="1_1_1524066830287",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test6",owner="1_1_1524042799129",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test5",owner="1_1_1523757594115",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test3",owner="1_1_1523696805278",zonegroup="0"} 36864.0
radosgw_usage_bucket_total{bucket="2test",owner="1_1_1524124857513",zonegroup="0"} 0.0
# HELP radosgw_usage_received_bytes_total Bytes received by the RADOSGW
# TYPE radosgw_usage_received_bytes_total counter
# HELP radosgw_usage_sent_bytes_total Bytes sent by the RADOSGW
# TYPE radosgw_usage_sent_bytes_total counter
# HELP radosgw_usage_ops_total Number of operations
# TYPE radosgw_usage_ops_total counter

@blemmenes
Copy link
Owner

@nanquanyuhao This is after you enabled RGW usage logging and generated some traffic?

I'll work on firing up a cluster using the demo container and see what I get.

Thanks,
Berant

@nanquanyuhao
Copy link
Author

@blemmenes No, this is the previous data...

@blemmenes
Copy link
Owner

@nanquanyuhao So have you configured 'rgw enable usage log = true'?

@nanquanyuhao
Copy link
Author

@blemmenes
As you say, my data like this when I had enabled the usage log...

# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 179806208.0
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 15282176.0
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1525112440.58
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.43000000000000005
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 7.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 65536.0
# HELP radosgw_usage_successful_ops_total Number of successful operations
# TYPE radosgw_usage_successful_ops_total counter
radosgw_usage_successful_ops_total{bucket="bucket_root",category="list_buckets",owner="anonymous"} 1.0
radosgw_usage_successful_ops_total{bucket="bucket_root",category="list_buckets",owner="testuser"} 3.0
radosgw_usage_successful_ops_total{bucket="my-new-bucket",category="create_bucket",owner="testuser"} 2.0
radosgw_usage_successful_ops_total{bucket="my-new-bucket",category="get_obj",owner="testuser"} 3.0
radosgw_usage_successful_ops_total{bucket="my-new-bucket",category="list_bucket",owner="testuser"} 2.0
radosgw_usage_successful_ops_total{bucket="my-new-bucket",category="put_acls",owner="testuser"} 4.0
radosgw_usage_successful_ops_total{bucket="my-new-bucket",category="put_obj",owner="testuser"} 4.0
radosgw_usage_successful_ops_total{bucket="your-new-bucket",category="create_bucket",owner="testuser"} 1.0
radosgw_usage_successful_ops_total{bucket="your-new-bucket",category="get_obj",owner="testuser"} 1.0
radosgw_usage_successful_ops_total{bucket="your-new-bucket",category="list_bucket",owner="testuser"} 1.0
radosgw_usage_successful_ops_total{bucket="your-new-bucket",category="put_acls",owner="testuser"} 2.0
radosgw_usage_successful_ops_total{bucket="your-new-bucket",category="put_obj",owner="testuser"} 2.0
# HELP radosgw_usage_bucket_total Bucket used bytes
# TYPE radosgw_usage_bucket_total gauge
radosgw_usage_bucket_total{bucket="my-new-bucket",owner="testuser",zonegroup="83d27dfb-c465-4762-9f7e-3e326a30764e"} 40960.0
radosgw_usage_bucket_total{bucket="your-new-bucket",owner="testuser",zonegroup="83d27dfb-c465-4762-9f7e-3e326a30764e"} 40960.0
# HELP radosgw_usage_received_bytes_total Bytes received by the RADOSGW
# TYPE radosgw_usage_received_bytes_total counter
radosgw_usage_received_bytes_total{bucket="bucket_root",category="list_buckets",owner="anonymous"} 0.0
radosgw_usage_received_bytes_total{bucket="bucket_root",category="list_buckets",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="my-new-bucket",category="create_bucket",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="my-new-bucket",category="get_obj",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="my-new-bucket",category="list_bucket",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="my-new-bucket",category="put_acls",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="my-new-bucket",category="put_obj",owner="testuser"} 73094.0
radosgw_usage_received_bytes_total{bucket="your-new-bucket",category="create_bucket",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="your-new-bucket",category="get_obj",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="your-new-bucket",category="list_bucket",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="your-new-bucket",category="put_acls",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="your-new-bucket",category="put_obj",owner="testuser"} 36547.0
# HELP radosgw_usage_sent_bytes_total Bytes sent by the RADOSGW
# TYPE radosgw_usage_sent_bytes_total counter
radosgw_usage_sent_bytes_total{bucket="bucket_root",category="list_buckets",owner="anonymous"} 214.0
radosgw_usage_sent_bytes_total{bucket="bucket_root",category="list_buckets",owner="testuser"} 957.0
radosgw_usage_sent_bytes_total{bucket="my-new-bucket",category="create_bucket",owner="testuser"} 0.0
radosgw_usage_sent_bytes_total{bucket="my-new-bucket",category="get_obj",owner="testuser"} 36559.0
radosgw_usage_sent_bytes_total{bucket="my-new-bucket",category="list_bucket",owner="testuser"} 1634.0
radosgw_usage_sent_bytes_total{bucket="my-new-bucket",category="put_acls",owner="testuser"} 0.0
radosgw_usage_sent_bytes_total{bucket="my-new-bucket",category="put_obj",owner="testuser"} 0.0
radosgw_usage_sent_bytes_total{bucket="your-new-bucket",category="create_bucket",owner="testuser"} 0.0
radosgw_usage_sent_bytes_total{bucket="your-new-bucket",category="get_obj",owner="testuser"} 12.0
radosgw_usage_sent_bytes_total{bucket="your-new-bucket",category="list_bucket",owner="testuser"} 819.0
radosgw_usage_sent_bytes_total{bucket="your-new-bucket",category="put_acls",owner="testuser"} 0.0
radosgw_usage_sent_bytes_total{bucket="your-new-bucket",category="put_obj",owner="testuser"} 0.0
# HELP radosgw_usage_ops_total Number of operations
# TYPE radosgw_usage_ops_total counter
radosgw_usage_ops_total{bucket="bucket_root",category="list_buckets",owner="anonymous"} 1.0
radosgw_usage_ops_total{bucket="bucket_root",category="list_buckets",owner="testuser"} 3.0
radosgw_usage_ops_total{bucket="my-new-bucket",category="create_bucket",owner="testuser"} 2.0
radosgw_usage_ops_total{bucket="my-new-bucket",category="get_obj",owner="testuser"} 3.0
radosgw_usage_ops_total{bucket="my-new-bucket",category="list_bucket",owner="testuser"} 2.0
radosgw_usage_ops_total{bucket="my-new-bucket",category="put_acls",owner="testuser"} 4.0
radosgw_usage_ops_total{bucket="my-new-bucket",category="put_obj",owner="testuser"} 4.0
radosgw_usage_ops_total{bucket="your-new-bucket",category="create_bucket",owner="testuser"} 1.0
radosgw_usage_ops_total{bucket="your-new-bucket",category="get_obj",owner="testuser"} 1.0
radosgw_usage_ops_total{bucket="your-new-bucket",category="list_bucket",owner="testuser"} 1.0
radosgw_usage_ops_total{bucket="your-new-bucket",category="put_acls",owner="testuser"} 2.0
radosgw_usage_ops_total{bucket="your-new-bucket",category="put_obj",owner="testuser"} 2.0

I still hope you will built some grafana dashboards specific to the metrics produced by this exporter as quickly as possible， thank you very much...

@blemmenes
Copy link
Owner

I apologize if I'm missing your issue but is this data not what you would expect? Now that you have enabled usage logging the above output looks normal to me.

@nanquanyuhao
Copy link
Author

@blemmenes
This data is what I need...
If enable the usage log , this data is what I need... 3Q..

@blemmenes
Copy link
Owner

@nanquanyuhao I'm afraid I'm misunderstanding what you are missing. This exporter only produces 5 different metrics and in your example above I show data from all 5.

Can you please show examples of what you expect to see that you do not?

Thanks,
Berant

@nanquanyuhao
Copy link
Author

@blemmenes
I am meaning that when I enable the usage log, this exporter can produces all the 5 different metrics...
Before I configuring it, I can't get the all 5 metrics but only radosgw_usage_bucket_total...

Very thanks...

@blemmenes
Copy link
Owner

Ah, in that case it is expected behavior. The exporter cannot show the usage metrics if the ceph RADOSGWs are not tracking and recording the usage information.

It is a requirement of this exporter (and any other tool that wants to get usage information out of the admin ops API) that rgw enable usage log = true is enabled.

Thanks,
Berant

@k0ste
Copy link
Contributor

k0ste commented Jun 21, 2018

Since Luminous 12.2.5 mgr was get rgw metrics support (ceph/ceph#21303) we can mix usage of this exporter and mgr/prometheus. With metadata we can get metrics of each rgw instance.

selection_002

@nanquanyuhao
Copy link
Author

@k0ste Thinks... In fact, what I need is the usage of bucket for user...
I have finish it like this:
default

Thinks ant way...

@k0ste
Copy link
Contributor

k0ste commented Jun 21, 2018

@nanquanyuhao bucket usage or "how much eat each user" is easy:

selection_002

@blemmenes
Copy link
Owner

blemmenes commented Jun 21, 2018 •
edited
Loading

@k0ste I've not really done any investigation of what the manager prometheus instrumentation exposes. I used ceph-nano to spin up a 12.2.5 cluster and this is what I got after creating a bucket and uploading one file to it.

# HELP ceph_rgw_qlen Queue length      
# TYPE ceph_rgw_qlen gauge             
ceph_rgw_qlen{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0           
# HELP ceph_rgw_req Requests           
# TYPE ceph_rgw_req counter            
ceph_rgw_req{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 5.0            
# HELP ceph_rgw_cache_miss Cache miss  
# TYPE ceph_rgw_cache_miss counter     
ceph_rgw_cache_miss{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 2.0     
ceph_objecter_op_rmw{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0    
# HELP ceph_rgw_put Puts               
# TYPE ceph_rgw_put counter            
ceph_rgw_put{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 1.0            
ceph_pool_metadata{pool_id="4",name="default.rgw.log"} 1.0                     
ceph_pool_metadata{pool_id="3",name="default.rgw.meta"} 1.0                    
ceph_pool_metadata{pool_id="5",name="default.rgw.buckets.index"} 1.0           
ceph_pool_metadata{pool_id="1",name=".rgw.root"} 1.0                           
ceph_pool_metadata{pool_id="2",name="default.rgw.control"} 1.0                 
ceph_pool_metadata{pool_id="6",name="default.rgw.buckets.data"} 1.0            
ceph_objecter_op_active{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0 
# HELP ceph_rgw_keystone_token_cache_hit Keystone token cache hits             
# TYPE ceph_rgw_keystone_token_cache_hit counter                               
ceph_rgw_keystone_token_cache_hit{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0                                                                      
# HELP ceph_rgw_metadata RGW Metadata  
# TYPE ceph_rgw_metadata untyped       
ceph_rgw_metadata{id="ceph-nano-luminous-faa32aebf00b",hostname="ceph-nano-luminous-faa32aebf00b",ceph_version="ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)"} 1.0
# HELP ceph_rgw_qactive Active requests queue                                  
# TYPE ceph_rgw_qactive gauge          
ceph_rgw_qactive{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0        
# HELP ceph_rgw_get_b Size of gets     
# TYPE ceph_rgw_get_b counter          
ceph_rgw_get_b{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0          
# HELP ceph_rgw_failed_req Aborted requests                                    
# TYPE ceph_rgw_failed_req counter     
ceph_rgw_failed_req{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 1.0     
# HELP ceph_rgw_keystone_token_cache_miss Keystone token cache miss            
# TYPE ceph_rgw_keystone_token_cache_miss counter                              
ceph_rgw_keystone_token_cache_miss{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0                                                                     
# HELP ceph_rgw_get Gets               
# TYPE ceph_rgw_get counter            
ceph_rgw_get{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0            
# HELP ceph_rgw_cache_hit Cache hits   
# TYPE ceph_rgw_cache_hit counter      
ceph_rgw_cache_hit{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 12.0     
ceph_objecter_op_w{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 1692.0   
ceph_objecter_op_r{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 337.0    
# HELP ceph_rgw_put_b Size of puts     
# TYPE ceph_rgw_put_b counter          
ceph_rgw_put_b{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 5469.0

I don't see any overlap with the metrics we are getting via the usage/bucket API though I could be missing something...

@k0ste
Copy link
Contributor

k0ste commented Jun 22, 2018

No really overlap, because radosgw_usage_exporter is focused on bucket usage/ops, mgr/prometheus is focused on rgw instance requests/data transmission.
So sum of ops/bytes per bucket we have is the one total metric of mgr, per instance.

@icoolchn
Copy link

Since Luminous 12.2.5 mgr was get rgw metrics support (ceph/ceph#21303) we can mix usage of this exporter and mgr/prometheus. With metadata we can get metrics of each rgw instance.

selection_002

hello, give me the mgr rgw grafana dashaboard ,thanks

@symbatkarzhassov
Copy link

https://github.com/ceph/ceph/tree/master/monitoring/grafana/dashboards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@blemmenes @nanquanyuhao @k0ste @icoolchn @symbatkarzhassov