Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

server returned HTTP status 404 Not Found #9366

Open
hunter-luo666 opened this issue Sep 7, 2024 · 3 comments · May be fixed by #9614
Open

server returned HTTP status 404 Not Found #9366

hunter-luo666 opened this issue Sep 7, 2024 · 3 comments · May be fixed by #9614
Labels
status: waiting-for-triage

Comments

@hunter-luo666
Copy link

Bug Report

Describe the bug

when we start fluent-bit , can not get prometheus metrics

root@ip-10-70-36-133:~# lsof -i:2021
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
fluent-bi 2164251 root 74u IPv4 19834897 0t0 TCP *:2021 (LISTEN)
fluent-bi 2164251 root 108u IPv4 19834995 0t0 TCP ip-10-70-36-133.ap-northeast-1.compute.internal:2021->ip-10-70-xx-xx.ap-northeast-1.compute.internal:43444 (ESTABLISHED)

To Reproduce

  • Rubular link if applicable:
  • Example log message if applicable:
{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}
  • Steps to reproduce the problem:

Expected behavior

Screenshots

Your Environment

Additional context

install service by run curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
root@ip-10-70-36-133:~# cat /etc/fluent-bit/fluent-bit.conf
[SERVICE]
flush 1
daemon Off
log_level info
parsers_file parsers.conf
plugins_file plugins.conf
http_server On
http_listen 0.0.0.0
http_port 2020
Health_Check On
HC_Errors_Count 5
HC_Retry_Failure_Count 5
HC_Period 5
storage.metrics on

[INPUT]
name tail
path /data/apache-shardingsphere/logs/.log
path_key log_path
tag sharding-services
[INPUT]
name tail
path /var/log/.log
path_key log_path
tag sharding-system

[INPUT]
Name exec
Tag prod-sharding
Interval_Sec 30
command pg_isready -h 10.70.xxx.xxx -p3307 -U sharding -d sharding_db

[INPUT]
Name proc
Proc_Name java
Interval_Sec 30
Interval_NSec 0
Fd true
Mem true
[INPUT]
name process_exporter_metrics
tag process_metrics
process_include_pattern java
scrape_interval 10

[OUTPUT]
name prometheus_exporter
match process_metrics
host 0.0.0.0
port 2021
add_label app sharding

[OUTPUT]
name loki
match *
host loki.xxx.com
port 443
tls on
labels job=fluentbit, env=prod, namespace=sharding, app=sharding-xxxx

root@ip-10-70-xxx:~# systemctl status fluent-bit
● fluent-bit.service - Fluent Bit
Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; enabled; preset: enabled)
Active: active (running) since Sat 2024-09-07 05:08:23 UTC; 10min ago
Docs: https://docs.fluentbit.io/manual/
Main PID: 2164251 (fluent-bit)
Tasks: 10 (limit: 37850)
Memory: 90.6M (peak: 132.7M)
CPU: 10min 31.477s
CGroup: /system.slice/fluent-bit.service
└─2164251 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.0] inotify_fs_add(): inode=6293396 watch_fd=2 name=/data/apa>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=3584 watch_fd=1 name=/var/log/alt>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=79203 watch_fd=2 name=/var/log/ap>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=3597 watch_fd=3 name=/var/log/aut>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=79146 watch_fd=4 name=/var/log/cl>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=78496 watch_fd=5 name=/var/log/cl>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=21 watch_fd=6 name=/var/log/dpkg.>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=79335 watch_fd=7 name=/var/log/fo>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=3083 watch_fd=8 name=/var/log/ker>
Sep 07 05:12:31 ip-10-70-36-133 fluent-bit[2164251]: [2024/09/07 05:12:31] [ info] [input:tail:tail.1] inotify_fs_add(): inode=3599 watch_fd=9 name=/var/log/ubu>
lines 1-21/21 (END)

root@ip-10-70-36-133:# curl http://10.70.36.133:2021/metrics
root@ip-10-70-36-133:# curl http://10.70.36.133:2021/metrics -I
HTTP/1.1 404 Not Found
Server: Monkey/1.7.2
Date: Sat, 07 Sep 2024 05:20:34 GMT
Transfer-Encoding: chunked

root@ip-10-70-36-133:# curl http://localhost:2021/metrics
root@ip-10-70-36-133:# curl http://localhost:2021/metrics -I
HTTP/1.1 404 Not Found
Server: Monkey/1.7.2
Date: Sat, 07 Sep 2024 05:20:59 GMT
Transfer-Encoding: chunked

root@ip-10-70-36-133:~#
@hunter-luo666
Copy link
Author

hunter-luo666 commented Sep 7, 2024 •
edited
Loading

image

we have CurrEstab(ESTABLISHED) 43680 , because of it is sharding-proxy instance

image

@patrick-stephens
Copy link
Contributor

Weird, seems to work fine with the node exporter example from the docs: https://docs.fluentbit.io/manual/pipeline/outputs/prometheus-exporter

Tried with both 3.1.3 and 3.1.7:

docker run --rm -it -p 2021:2021 -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro fluent/fluent-bit:3.1.7
...
curl localhost:2021/metrics                                                                                        
# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="idle"} 8048.1099999999997
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="iowait"} 5.5099999999999998
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="irq"} 0
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="nice"} 0
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="softirq"} 120.58
node_cpu_seconds_total{app="fluent-bit",color="blue",cpu="0",mode="steal"} 0

Can you use log_level debug and check if any warnings or errors reported for opening the port or similar?

What actual Ubuntu package did you install? I've no idea what EC stuff maps to.

@jplitza
Copy link

jplitza commented Nov 18, 2024

I'm seeing the same behavior when running with a log_to_metrics filter: The prometheus_exporter seems to return 404 until the first matching log comes by. As long as no metrics are delivered to the output, the endpoint returns 404.

@jplitza jplitza linked a pull request Nov 18, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status: waiting-for-triage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

out_prometheus_exporter: Return 204 on no metrics jplitza/fluent-bit
3 participants
@jplitza @patrick-stephens @hunter-luo666