services started with docker compose won't be stopped by sablier unless hit with a request first #153

xsolinsx · 2023-05-09T08:29:28Z

Describe the bug
I have a docker compose which starts a multitude of services, sablier is correctly configured as it is able to start/stop services as needed but (here's the bug) ONLY when the services have been hit by a request.
If I only start the services with docker compose up -d all services would stay up & running forever.
In order for them to be "detected" by sablier it seems I need to hit them with a request.

Context

Sablier version: 1.3.0
Provider: docker 23.0.5
Reverse proxy: traefik 2.10.1
Sablier running inside a container? YES

Expected behavior
After starting services with docker compose up -d Sablier should wait for the time configured in the config and then stop them without the need to hit them first with a request.

The text was updated successfully, but these errors were encountered:

acouvreur · 2023-05-09T13:04:48Z

Hi @xsolinsx,

Without any prior request, Sablier has no way of knowing which containers to start and stop.

The beta feature of group using autodiscovery of containers with sablier labels could implement such feature.

xsolinsx · 2023-05-09T14:45:06Z

Nice, so this is something that will be implemented in the next release right?
I will leave this open as I do not know whether you want to close this right now or when you release the next version, I will leave the decision to you 😄
Thanks

Edit: I've found a sketchy workaround to execute right after the docker compose up -d
Grep the line with the names of the containers, get only the container(s) names for each middleware, replace comma if present, reduce to one line and stop the containers

grep -h "names: " /path/to/traefik/file_dynamic_conf/sablier_middlewares_definitions.yml |
    cut -d':' -f 2 |
    sed "s/,/ /g" |
    tr '\n' ' ' |
    xargs docker stop

IzeQube · 2023-06-20T11:27:30Z

I have just started implementing sablier with caddy and the new groups function.

Do you think it would be possible to include the function to shutdown not needed containers if they are up, but shouldn't be up because of missing requests...

ab623 · 2023-07-14T16:25:57Z

Hi @xsolinsx,

Without any prior request, Sablier has no way of knowing which containers to start and stop.

The beta feature of group using autodiscovery of containers with sablier labels could implement such feature.

On startup why cant sablier get a list of all containers with the sablier.enable label set to true, and apply the default session time to them?

I've also noticed that even through the service is up, once sablier first starts, it still shows the "waiting" screen on the first hit, even though it should bypass it as the service is up. I assume its for the same underlying reason, it hasnt built this "internal registry" of services.

webysther · 2023-07-18T22:23:50Z

Add healthcheck to you services fix the problem or configure a status server to check.

ab623 · 2023-07-19T08:58:32Z

There already is a healthcheck on the service.

Currently

Service A comes up (showing healthy)
Sablier comes up
Wait for an hour
Service A is still up

Expected

Service A comes up (showing healthy)
Sablier comes up
Wait for an hour
Sablier brings down Service A

Add healthcheck to you services fix the problem or configure a status server to check.

What is a status server?

acouvreur · 2023-07-19T13:21:48Z

Sablier does not (yet) pick up already started services, you need to access them at least once in order for them to be downscaled.

I will probably change this behavior with an option in the future.

webysther · 2023-07-19T16:51:21Z

There already is a healthcheck on the service.

Currently

Service A comes up (showing healthy)

Sablier comes up

Wait for an hour

Service A is still up

Expected

Service A comes up (showing healthy)

Sablier comes up

Wait for an hour

Sablier brings down Service A

Add healthcheck to you services fix the problem or configure a status server to check.

What is a status server?

like uptime kuma, but, nevermind, I was wrong about it.

github-actions · 2024-02-07T01:57:50Z

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

xsolinsx · 2024-02-07T08:49:27Z

unstale

github-actions · 2024-03-10T02:03:36Z

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

xsolinsx · 2024-03-10T19:16:56Z

unstale

MundMBo · 2024-03-15T08:34:10Z

I am also quite interested in the resolution of this bug. Is there anything planned (roughly) when it will be solved? Will it be part of the next release?

My current workaround for this bug is to extend my CICD-pipeline. After deploying and starting the container I am sending one request to the container without waiting for any response.

curl --head --max-time 1 "https://route.to/docker/service" >/dev/null 2>&1

This is working for the dynamic and blocking strategy of sablier, as well as for all my containers regardless of their response and the http response status code.

ab623 · 2024-03-15T10:58:10Z

I do the same. When I restart services I have to hit them once with a request so Sablier kicks in. Not ideal.

acouvreur · 2024-07-03T19:08:48Z

Reviving this issue, I will create this feature but I'd like to gather some feedback here.

The goal is to create a startup behavior after everything has been loaded properly, workloads that have been auto-discovered (using labels), will be shutdown if they are currently running and not in an active session.

How would you name such option ?

auto-stop
auto-stop-on-startup
shutdown-running-discovered-workloads
etc.

if you have some ideas, that'd be great. I know that @gorositopablo you'd be interested by this.

IzeQube · 2024-07-03T19:11:40Z

I think I like auto-stop-on-startup.

And I am really looking forward to this new function.

This feature adds the capability to stop unregistered running instances upon startup. Previously, you had to stop running instances manually or issue an initial request that will shut down instances afterwards. With this change, all discovered instances will be shutdown. They need to be registered using labels. E.g.: sablier.enable=true Fixes #153

pablogorosito · 2024-07-04T08:23:44Z

Agree, I believe that auto-stop-on-startup would be better.

This feature adds the capability to stop unregistered running instances upon startup. Previously, you had to stop running instances manually or issue an initial request that will shut down instances afterwards. With this change, all discovered instances will be shutdown. They need to be registered using labels. E.g.: sablier.enable=true Fixes #153

acouvreur · 2024-07-04T16:10:01Z

🎉 This issue has been resolved in version 1.8.0-beta.7 🎉

The release is available on:

v1.8.0-beta.7
GitHub release

Your semantic-release bot 📦🚀

acouvreur · 2024-07-04T17:07:20Z

You can try using the docker tag 1.8.0-beta.7.

This is enabled by default. Because as you all stated, this is the behavior we'd actually expect from Sablier. And I agree.

It's not a breaking change in the way that old configurations are still valid. You can opt-out by setting provider.auto-stop-on-startup to false.

I'm waiting for your feedback!

valankar · 2024-07-04T18:20:06Z

Thanks for implementing this. Just a quick question. If I'm using names in my Caddyfile, I also need to set the label sablier.enable=true in the separate containers, correct?

acouvreur · 2024-07-04T18:47:19Z

Thanks for implementing this. Just a quick question. If I'm using names in my Caddyfile, I also need to set the label sablier.enable=true in the separate containers, correct?

Yes.

Sablier needs to know which container to stop. For that, you need to add the following label sablier.enable=true.

You can still use names to target containers, there's no issue with that.

This is purely auto-discovery done by Sablier at startup, and it does not involve any reverse proxy configuration or adjustment.

valankar · 2024-07-04T18:51:01Z

Just tested with the beta and it works well.

xsolinsx · 2024-07-04T21:59:12Z

if I run twice the same docker compose up -d sablier only kills the containers the first time, not the second one

deimosfr · 2024-07-05T00:58:29Z

I confirm I have the same behavior

acouvreur · 2024-07-05T01:13:40Z

if I run twice the same docker compose up -d sablier only kills the containers the first time, not the second one

I confirm I have the same behavior

Can you please share a quick reproducible compose file ?
Also, with the debug log enabled, what do you see in the logs ?

One behavior of the auto-stop is to stop all containers that are currently running, but that Sablier is not aware of.
By default, Sablier will save the current state upon shutdown.

So upon restart, Sablier loads back your containers session from the previous run and it will not shut them down.
You can try by disabling storage completely, I think this issue won't happen again.

valankar · 2024-07-05T01:32:36Z

I think what they are referring to is the auto-stopping is only run at startup and not periodically. Then, if you restart your container (not the sablier one), it does not get stopped again until you make another request. i.e. The containers are stopped at "startup" of sablier only.

Perhaps it should run periodically?

acouvreur · 2024-07-05T01:50:50Z

I think what they are referring to is the auto-stopping is only run at startup and not periodically. Then, if you restart your container (not the sablier one), it does not get stopped again until you make another request. i.e. The containers are stopped at "startup" of sablier only.

Perhaps it should run periodically?

The current behavior is to stop the found containers at startup only, not periodically.

I think that the desired behavior is actually to register all your containers, and have Sablier stop them for you.
If you start the container manually while Sablier is running, it will not stop it again for you.

If the current behavior does not meet your needs, maybe we can refine, change or add configuration option to get to the desired behavior.

What behaviors would you like to have ?

valankar · 2024-07-05T06:53:00Z

For my case, I'm happy with the current behavior. However, I can imagine the scenario described in:

https://github.com/acouvreur/sablier/issues/153#issuecomment-1999167626

For example, say you have some automatic updates of your container, or you are doing some development requiring updating it. You may be only hitting the service on an internal address and not via Sablier. Every time that happens, you'd need to access it with Sablier to have it notice the new up state.

You can of course update the image of your containers while they are stopped, which is one way around this. Perhaps others can chime in on their use case/reasoning.

deimosfr · 2024-07-05T08:57:38Z

I think what they are referring to is the auto-stopping is only run at startup and not periodically. Then, if you restart your container (not the sablier one), it does not get stopped again until you make another request. i.e. The containers are stopped at "startup" of sablier only.

Perhaps it should run periodically?

Exactly the issue and behavior I expect! Check periodically would be the solution from my POV.

valankar · 2024-07-05T09:18:54Z

I think doing it periodically makes sense, though we should be a bit careful on the period. Imagine a scenario where you bring up this container at the end of the time period. It will immediately get stopped.

It should probably work like:

Sablier detects that a container is up that should not be.
Sablier resets its timer for the service just as if a request has been made.
Container is shutdown when the timer is up.

acouvreur · 2024-07-06T16:05:19Z

Perhaps this feature should be some kind of "reconciliation" that could happen periodically.

I better understand the use case now.

We can maybe add this as an extra feature, and also have some opt-in/opt-out labels.

We can go further and have some labels with pre-defined hours for reconciliation. Any way, we might have w new feature here, which is not the auto-stop-on-startup.

From my point of view, the startup part of auto-stop-on-startup was referring to Sablier. What do you think ?

gorositopablo · 2024-07-08T14:04:43Z

I'm not sure about all the comments with docker-compose, but I've tried with sablier:1.8.0-beta.7 in Kubernetes with Traefik and the containers are only terminated at Sablier boot up.

So, let's say that I:
1- Deploy Sablier
2- Deploy my workload (using labels)
3- Nothing happens
4a- restarting Sablier it kill the pod, or (new behavior with sablier:1.8.0-beta.7)
4b- making a request to the backend, it kill the pod. (classic behavior)

While Sablier does kill the pod at the start of its lifecycle, it means that I need to restart Sablier each time that I have a new release.

Thanks @acouvreur

deimosfr · 2024-07-11T06:13:20Z

Good Idea @acouvreur . I think we can start simple without specific hours and add it later if requested? I personaly don't have this need

acouvreur · 2024-10-02T00:36:29Z

🎉 This issue has been resolved in version 1.8.0 🎉

The release is available on:

v1.8.0
GitHub release

Your semantic-release bot 📦🚀

xsolinsx added the bug Something isn't working label May 9, 2023

ab623 mentioned this issue Aug 8, 2023

feat(api) Added Status API #167

Closed

github-actions bot added the Stale label Feb 7, 2024

github-actions bot removed the Stale label Feb 9, 2024

github-actions bot added the Stale label Mar 10, 2024

github-actions bot removed the Stale label Mar 11, 2024

acouvreur mentioned this issue Jul 3, 2024

Feature Request: Automatically create and delete a horizontal pod autoscaler #130

Closed

acouvreur linked a pull request Jul 4, 2024 that will close this issue

feat(providers): add provider.auto-stop-on-startup argument #346

Merged

acouvreur added the released on @beta label Jul 4, 2024

acouvreur closed this as completed in dfb9bac Oct 2, 2024

acouvreur added the released label Oct 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

services started with docker compose won't be stopped by sablier unless hit with a request first #153

services started with docker compose won't be stopped by sablier unless hit with a request first #153

xsolinsx commented May 9, 2023

acouvreur commented May 9, 2023

xsolinsx commented May 9, 2023 •

edited

Loading

IzeQube commented Jun 20, 2023

ab623 commented Jul 14, 2023

webysther commented Jul 18, 2023 •

edited

Loading

ab623 commented Jul 19, 2023 •

edited

Loading

acouvreur commented Jul 19, 2023 •

edited

Loading

webysther commented Jul 19, 2023

github-actions bot commented Feb 7, 2024

xsolinsx commented Feb 7, 2024

github-actions bot commented Mar 10, 2024

xsolinsx commented Mar 10, 2024

MundMBo commented Mar 15, 2024

ab623 commented Mar 15, 2024

acouvreur commented Jul 3, 2024 •

edited

Loading

IzeQube commented Jul 3, 2024

pablogorosito commented Jul 4, 2024

acouvreur commented Jul 4, 2024

acouvreur commented Jul 4, 2024 •

edited

Loading

valankar commented Jul 4, 2024

acouvreur commented Jul 4, 2024

valankar commented Jul 4, 2024

xsolinsx commented Jul 4, 2024

deimosfr commented Jul 5, 2024

acouvreur commented Jul 5, 2024 •

edited

Loading

valankar commented Jul 5, 2024

acouvreur commented Jul 5, 2024

valankar commented Jul 5, 2024

deimosfr commented Jul 5, 2024

valankar commented Jul 5, 2024 •

edited

Loading

acouvreur commented Jul 6, 2024

gorositopablo commented Jul 8, 2024

deimosfr commented Jul 11, 2024

acouvreur commented Oct 2, 2024

services started with docker compose won't be stopped by sablier unless hit with a request first #153

services started with docker compose won't be stopped by sablier unless hit with a request first #153

Comments

xsolinsx commented May 9, 2023

acouvreur commented May 9, 2023

xsolinsx commented May 9, 2023 • edited Loading

IzeQube commented Jun 20, 2023

ab623 commented Jul 14, 2023

webysther commented Jul 18, 2023 • edited Loading

ab623 commented Jul 19, 2023 • edited Loading

acouvreur commented Jul 19, 2023 • edited Loading

webysther commented Jul 19, 2023

github-actions bot commented Feb 7, 2024

xsolinsx commented Feb 7, 2024

github-actions bot commented Mar 10, 2024

xsolinsx commented Mar 10, 2024

MundMBo commented Mar 15, 2024

ab623 commented Mar 15, 2024

acouvreur commented Jul 3, 2024 • edited Loading

IzeQube commented Jul 3, 2024

pablogorosito commented Jul 4, 2024

acouvreur commented Jul 4, 2024

acouvreur commented Jul 4, 2024 • edited Loading

valankar commented Jul 4, 2024

acouvreur commented Jul 4, 2024

valankar commented Jul 4, 2024

xsolinsx commented Jul 4, 2024

deimosfr commented Jul 5, 2024

acouvreur commented Jul 5, 2024 • edited Loading

valankar commented Jul 5, 2024

acouvreur commented Jul 5, 2024

valankar commented Jul 5, 2024

deimosfr commented Jul 5, 2024

valankar commented Jul 5, 2024 • edited Loading

acouvreur commented Jul 6, 2024

gorositopablo commented Jul 8, 2024

deimosfr commented Jul 11, 2024

acouvreur commented Oct 2, 2024

xsolinsx commented May 9, 2023 •

edited

Loading

webysther commented Jul 18, 2023 •

edited

Loading

ab623 commented Jul 19, 2023 •

edited

Loading

acouvreur commented Jul 19, 2023 •

edited

Loading

acouvreur commented Jul 3, 2024 •

edited

Loading

acouvreur commented Jul 4, 2024 •

edited

Loading

acouvreur commented Jul 5, 2024 •

edited

Loading

valankar commented Jul 5, 2024 •

edited

Loading