-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Update Altacv * Update hugo * Build on PR
- Loading branch information
1 parent
6883e4b
commit 73e507a
Showing
8 changed files
with
274 additions
and
15 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
--- | ||
title: About | ||
date: 2024-12-26T07:47:40+01:00 | ||
draft: false | ||
toc: false | ||
--- | ||
|
||
Hey there! I'm Morteza, a software engineer who loves to deal with a new challenge every day. | ||
|
||
In case you're wondering, here is my [Resume](/cv.pdf) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,193 @@ | ||
--- | ||
title: Dockhand | ||
description: Restarding unhealthy docker containers | ||
summary: | | ||
If you have bunch od docker containers that go unhealthy and you want to restart them automatically, you have | ||
two options: either use deunhealth or use a simple bash script to do the same thing. | ||
date: 2024-12-26T07:47:40+01:00 | ||
draft: false | ||
toc: false | ||
images: | ||
keywords: | ||
- docker | ||
- healthcheck | ||
slug: dockhand | ||
tags: | ||
- docker | ||
categories: | ||
- Provisioning and orchestration | ||
--- | ||
|
||
## TL;DR | ||
|
||
Either use [deunhealth](https://github.com/qdm12/deunhealth) or run this bash script: | ||
|
||
{{< gist mortezaPRK db9f82fabcb4b5c441c89ab80e02a670 dockhand.sh >}} | ||
|
||
Now any container that has the label `autoheal=true` will be restarted when it goes unhealthy. | ||
|
||
|
||
## The problem | ||
|
||
I have bunch of docker containers running here and there. Sometimes, they go unhealthy and I have to restart them. But I don't want to do this manually. I want to automate this process. | ||
|
||
At first, I thought there should already be a way to do this and I found this niche tool called [deunhealth](https://github.com/qdm12/deunhealth). | ||
|
||
It was easy to install and use. All I had to do was to give access to the docker socket and add a specific label to the containers that I want to monitor. | ||
|
||
|
||
> [!CAUTION] | ||
> :loudspeaker: Giving access to the docker socket is a security risk! | ||
I'm paranoid about security and I don't want to give access to the docker socket to any random tool. No hard feelings, deunhealth :melting_face: | ||
|
||
> The reason is, I don't want to review every line of code in the tool to make sure it's not doing anything malicious, let alone checking every release and dependency! | ||
|
||
## The solution | ||
|
||
So what are my options? | ||
|
||
1. Don't be paranoid and use deunhealth (You can do this if you trust the tool, or you're a normal person) | ||
2. Write a simple script that does the same thing in Go or Python (since Docker has SDKs for these languages) | ||
3. Write a bash script that does the same thing | ||
|
||
|
||
For me, the first option is out of the question. | ||
|
||
The second option was easy to implement. Checking the [docker-py documentation](https://docker-py.readthedocs.io/en/stable/client.html), There is a method to listen to events: | ||
|
||
```python | ||
# pip install docker | ||
import docker | ||
|
||
client = docker.from_env() | ||
for event in client.events(decode=True): | ||
print(event) | ||
``` | ||
|
||
But this would print all events. I only want the events that: | ||
1. Are related to `health` of the container, preferably narrowed down to `unhealthy` status | ||
2. Are related to the containers only (not images, networks, etc.) | ||
3. Only emited for the containers that have a specific label (e.g. `autoheal=true`) | ||
|
||
### Jumping into the rabbit hole | ||
|
||
How can I test these filters? Docker cli already has a command to listen to the events: `docker system events`. | ||
|
||
Next, I want to run a container with a specific label and a simple healthcheck command to be able to see the events. | ||
|
||
```shell | ||
docker run --rm \ | ||
--name autohealtest \ | ||
--label 'autoheal=true' \ # The label that I want to filter the events with | ||
--health-cmd='test ! -f /tmp/hc || exit 1' \ # The healthcheck. It will fail if /tmp/hc exists | ||
--health-interval=10s \ | ||
--health-timeout=1s \ | ||
--health-retries=3 \ | ||
alpine:latest tail -f /dev/null # The command that will keep the container running forever | ||
``` | ||
|
||
In another terminal, I run the docker events command: | ||
|
||
```shell | ||
docker system events --format json | ||
|
||
# Output: | ||
# { | ||
# "status": "create", | ||
# "id": "9721b76dba119e3d778a5d62dca8f0329b4a3306672da19e4048f2c31e7e3f11", | ||
# "from": "alpine:latest", | ||
# "Type": "container", | ||
# "Action": "create", | ||
# "Actor": { | ||
# "ID": "9721b76dba119e3d778a5d62dca8f0329b4a3306672da19e4048f2c31e7e3f11", | ||
# "Attributes": { | ||
# "autoheal": "true", | ||
# "image": "alpine:latest", | ||
# "name": "autohealtest" | ||
# } | ||
# }, | ||
# "scope": "local", | ||
# "time": 1735191091, | ||
# "timeNano": 1735191091153980331 | ||
# } | ||
``` | ||
|
||
Now let's [filter the events](https://docs.docker.com/reference/cli/docker/system/events/#filter) by: | ||
1. Type: container | ||
2. Label: autoheal=true | ||
3. Health status: unhealthy | ||
|
||
```shell | ||
docker system events \ | ||
--filter type=container \ # Only container events | ||
--filter label=autoheal=true \ # Only the containers with the label autoheal=true | ||
--filter event='health_status: unhealthy' \ # Only the unhealthy containers | ||
--format json | ||
``` | ||
|
||
Now if I create a file in the `autohealtest` container under `/tmp/hc`, the healthcheck will fail and the container will be marked as unhealthy. | ||
|
||
```shell | ||
docker exec autohealtest touch /tmp/hc | ||
``` | ||
|
||
And I see the event in my terminal: | ||
```json | ||
{ | ||
"status": "health_status: unhealthy", | ||
"id": "19ccabc70b9011ad4ca4b67eb1b335ac3523d0b1174d238599a132085fdcb142", | ||
"from": "alpine:latest", | ||
"Type": "container", | ||
"Action": "health_status: unhealthy", | ||
"Actor": { | ||
"ID": "19ccabc70b9011ad4ca4b67eb1b335ac3523d0b1174d238599a132085fdcb142", | ||
"Attributes": { | ||
"autoheal": "true", | ||
"image": "alpine:latest", | ||
"name": "autohealtest" | ||
} | ||
}, | ||
"scope": "local", | ||
"time": 1735191438, | ||
"timeNano": 1735191438145468746 | ||
} | ||
``` | ||
|
||
### Back to Python | ||
|
||
Now that I know the filters, I can implement the Python script: | ||
|
||
```python | ||
# pip install docker | ||
import docker | ||
|
||
client = docker.from_env() | ||
for event in client.events( | ||
filters={ | ||
'type': 'container', | ||
'label': 'autoheal=true', | ||
'event': 'health_status: unhealthy' | ||
}, | ||
decode=True | ||
): | ||
client.containers.get(event['id']).restart() | ||
``` | ||
|
||
That's it! | ||
|
||
### The bash script | ||
|
||
As you already saw in the previous section, during this investigation I accidentally wrote the third option. | ||
|
||
A bash script that does the same thing + a systemd service to run it. | ||
|
||
{{< gist mortezaPRK db9f82fabcb4b5c441c89ab80e02a670 >}} | ||
|
||
|
||
## What's next? | ||
|
||
1. Blindly restarting the containers might be ok for simple scenarios, but I might need to add some checks to make sure the container is really unhealthy before restarting it, and maybe add a jitter before restarting several containers at the same time to avoid thundering herd problem. | ||
2. This can definitely be a good candidate for the next Docker or Compose feature. I'll open an issue on their GitHub (probably someone already did, I should've check that first :sweat_smile:) | ||
3. I heard about Docker plugins. Last time I checked, there was no way to manipulate containers, but I'll check again. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,24 +4,65 @@ title: Mortz's Corner | |
theme: hello-friend-ng | ||
copyright: Content licensed under CC0. | ||
enableGitInfo: true | ||
enableEmoji: true | ||
enableRobotsTXT: true | ||
disableRSS: true | ||
disableHugoGeneratorInject: true | ||
sectionPagesMenu: main | ||
|
||
module: | ||
mounts: | ||
- source: content | ||
target: content | ||
|
||
permalinks: | ||
posts: /posts/:year/:month/:title/ | ||
|
||
params: | ||
enableSharingButtons: true | ||
logo: | ||
logoMark: ">" | ||
logoText: Resume | ||
logoHomeLink: /cv.pdf | ||
logoText: Home | ||
logoHomeLink: / | ||
footer: | ||
trademark: 2024 | ||
trademark: 2025 | ||
rss: false | ||
copyright: true | ||
bottomText: | ||
- Powered by <a href="http://gohugo.io">Hugo</a> | ||
homeSubtitle: Comming soon! | ||
enableSharingButtons: true | ||
homeSubtitle: | | ||
Latest Posts | ||
{{ .Posts }} | ||
social: | ||
- name: email | ||
url: mailto:[email protected] | ||
- name: github | ||
url: https://github.com/mortezaPRK/ | ||
- name: linkedin | ||
url: https://www.linkedin.com/in/mortzprk/ | ||
url: https://www.linkedin.com/in/mortzprk/ | ||
|
||
privacy: | ||
disqus: | ||
disable: true | ||
googleAnalytics: | ||
disable: true | ||
instagram: | ||
disable: true | ||
twitter: | ||
disable: true | ||
vimeo: | ||
disable: true | ||
youtube: | ||
disable: true | ||
|
||
taxonomies: | ||
tag: tags | ||
category: categories | ||
|
||
menu: | ||
main: | ||
- identifier: about | ||
name: About | ||
url: /about/ |
Submodule altacv
updated
9 files
+25 −0 | CHANGELOG.md | |
+4 −19 | README.md | |
+99 −51 | altacv.cls | |
+ − | mmayer.pdf | |
+ − | mmayer.png | |
+10 −7 | mmayer.tex | |
+ − | sample.pdf | |
+ − | sample.png | |
+9 −5 | sample.tex |
Submodule hello-friend-ng
updated
13 files
+2 −2 | README.md | |
+6 −6 | assets/scss/_fonts.scss | |
+6 −0 | docs/svgs.md | |
+18 −8 | exampleSite/config.toml | |
+3 −3 | layouts/_default/list.html | |
+1 −1 | layouts/index.html | |
+1 −1 | layouts/partials/footer.html | |
+11 −6 | layouts/partials/head.html | |
+2 −2 | layouts/partials/header.html | |
+2 −2 | layouts/partials/menu.html | |
+1 −1 | layouts/partials/social-icons.html | |
+12 −0 | layouts/partials/svg.html | |
+1 −1 | layouts/posts/single.html |