Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory #2919

Open
atlasloewenherz opened this issue Mar 27, 2024 · 0 comments
Open

level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory #2919

atlasloewenherz opened this issue Mar 27, 2024 · 0 comments

Comments

@atlasloewenherz
Copy link

Hi all,

i spent the last two days trying to figure out what could be wrong with my installation and i could not figure it out and i need some help identifying the root cause of the the above mentioned problem maybe someon could spot something i can see!!

These are the steps i used to reproduce the above error:

environment

  • Gitlab 16.10.0
  • Postgresql 13.14
  • Ubuntu 22.04
  • Containerd 1.6.28
  • nerdctl 1.7.5

steps to reproduce

  1. created a database / user / credentials in a remote postgresql 13
  2. i created a bash script(*1) to start the gitlab instance using a .env file with env-from-file , as i try to avoid compose style of docker
  3. i started a local redis container which exposes its redis port to the localhost (lets say 10.10.2.8:6789 )
  4. in the gitlab environments i configured redis, postgresql and an openid connect authentication
  5. i started the gitlab container instance which start and even claim
Loading /etc/docker-gitlab/runtime/env-defaults
Initializing logdir...
Initializing datadir...
Container TimeZone -> Europe/Berlin
Installing configuration templates...
Configuring gitlab...
Configuring gitlab::database
Configuring /home/git/.postgresqlrc to avoid version mismatch on dumping
- Detected server version: 130014
- Generating /home/git/.postgresqlrc
13 10.10.2.5:5432 gitdb
- Uninstalling unused client(s): postgresql-client-14 postgresql-client-15 postgresql-client-16
Configuring gitlab::redis
Configuring gitlab::actioncable
Configuring gitlab::secrets...
Configuring gitlab::sidekiq...
Configuring gitlab::gitaly...
Configuring gitlab::monitoring...
Configuring gitlab::gitlab-workhorse...
Configuring gitlab::puma...
Configuring gitlab::timezone...
Configuring gitlab::rack_attack...
 Validating RACK_ATTACK_WHITELIST...
  input=127.0.0.1, to_range=127.0.0.1..127.0.0.1
Configuring gitlab::ci...
Configuring gitlab::artifacts...
Configuring gitlab::packages...
Configuring gitlab::terraform_state...
Configuring gitlab::lfs...
Configuring gitlab::uploads...
Configuring gitlab::mattermost...
Configuring gitlab::project_features...
Configuring gitlab::smtp_settings...
Configuring gitlab::oauth...
Configuring gitlab::oauth::generic...
Configuring gitlab::ldap...
Configuring gitlab::cron_jobs...
Configuring gitlab::backups...
Configuring gitlab::backups::schedule...
Configuring gitlab::registry...
Configuring gitlab::pages...
Configuring gitlab::sentry...
Configuring gitlab::content_security_policy...
Configuring gitlab-shell...
Configuring nginx...
Configuring nginx::gitlab...
Clearing cache...
2024-03-27 03:52:29,451 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2024-03-27 03:52:29,451 INFO Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing
2024-03-27 03:52:29,451 INFO Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing
2024-03-27 03:52:29,451 INFO Included extra file "/etc/supervisor/conf.d/groups.conf" during parsing
2024-03-27 03:52:29,451 INFO Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing
2024-03-27 03:52:29,451 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2024-03-27 03:52:29,451 INFO Included extra file "/etc/supervisor/conf.d/puma.conf" during parsing
2024-03-27 03:52:29,451 INFO Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
2024-03-27 03:52:29,451 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2024-03-27 03:52:29,451 INFO Set uid to user 0 succeeded
2024-03-27 03:52:29,455 INFO RPC interface 'supervisor' initialized
2024-03-27 03:52:29,455 INFO supervisord started with pid 1512
2024-03-27 03:52:30,457 INFO spawned: 'gitaly' with pid 1519
2024-03-27 03:52:30,459 INFO spawned: 'puma' with pid 1520
2024-03-27 03:52:30,460 INFO spawned: 'gitlab-workhorse' with pid 1521
2024-03-27 03:52:30,461 INFO spawned: 'sidekiq' with pid 1522
2024-03-27 03:52:30,462 INFO spawned: 'sshd' with pid 1523
2024-03-27 03:52:30,463 INFO spawned: 'nginx' with pid 1524
2024-03-27 03:52:32,087 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-27 03:52:32,087 INFO success: puma entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-27 03:52:32,087 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-27 03:52:32,087 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-27 03:52:32,087 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-27 03:52:32,087 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-27 03:52:32,087 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

the logs look almost beautifull! except that when i try to access the web frontend i get a 502 waiting for gitlab to boot and unfrotunately the instance does proceed further so i checked the logs one by one, the only logs files that seem to tell me that there is something wrong with puma is gitlab-workhorst.log file which shows the following:

time="2024-03-27T04:04:40+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:04:50+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:05:00+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:05:10+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:05:20+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:05:30+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:05:40+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:05:50+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:06:00+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:06:10+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:06:20+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:06:30+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:06:40+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:06:50+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:07:00+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:07:10+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:07:20+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:07:30+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:07:40+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:07:50+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:08:00+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:08:10+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:08:20+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:08:30+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:08:40+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:08:50+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:09:00+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=
time="2024-03-27T04:09:10+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /home/git/gitlab/tmp/sockets/gitlab.socket: connect: no such file or directory" method=GET uri=

so i checked the container to see if the socket is created or not:

➜  conf.d nerdctrl exec -it gitlab ls  -al /home/git/gitlab/tmp/sockets/
total 20
drwxrwxr-x 1 git git 4096 Mar 21 22:44 .
drwxrwxr-x 1 git git 4096 Mar 21 22:56 ..
-rw-rw-r-- 1 git git    0 Mar 21 22:44 .gitkeep
drwxrwxr-x 1 git git 4096 Mar 27 03:52 private
➜  conf.d

so it seems the socket is not there lets now check the puma logs:

{"timestamp":"2024-03-27T02:52:34.452Z","pid":1520,"message":"Puma starting in cluster mode..."}
{"timestamp":"2024-03-27T02:52:34.477Z","pid":1520,"message":"* Puma version: 6.4.0 (ruby 3.2.3-p157) (\"The Eagle of Durango\")"}
{"timestamp":"2024-03-27T02:52:34.477Z","pid":1520,"message":"*  Min threads: 1"}
{"timestamp":"2024-03-27T02:52:34.478Z","pid":1520,"message":"*  Max threads: 16"}
{"timestamp":"2024-03-27T02:52:34.485Z","pid":1520,"message":"*  Environment: production"}
{"timestamp":"2024-03-27T02:52:34.486Z","pid":1520,"message":"*   Master PID: 1520"}
{"timestamp":"2024-03-27T02:52:34.486Z","pid":1520,"message":"*      Workers: 3"}
{"timestamp":"2024-03-27T02:52:34.486Z","pid":1520,"message":"*     Restarts: (<E2><9C><94>) hot (<E2><9C><96>) phased"}
{"timestamp":"2024-03-27T02:52:34.486Z","pid":1520,"message":"* Preloading application"}

im able to reproduce this behaviour with the above configuration.

Thanks for you help!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@atlasloewenherz