mikkmokk-proxy is an unobtrusive reverse proxy server that injects faults on the HTTP layer.
Have you ever wanted to test how well your (backend|frontend) services handles
- failed requests
- duplicate requests
- delayed requests
mikkmokk-proxy is — literally — a gateway drug for resiliency testing on the HTTP layer.
sequenceDiagram
client->>mikkmokk: POST
mikkmokk->>mikkmokk: 💤delay before?💤
mikkmokk->>client: 💥fail before?💥
mikkmokk->>destination: POST
mikkmokk->>destination: 💥duplicate POST?💥
destination->>database: read and/or state change
database->>destination: result
destination->>mikkmokk: result
mikkmokk->>mikkmokk: 💤delay after?💤
mikkmokk->>client: result OR 💥fail after?💥
client here is anything that will normally access destination
using the HTTP protocol, but goes via mikkmokk instead.
mikkmokk can inject five different types of faults:
- fail a request before the destination is reached
- fail a request after the destination is reached
- add a delay before accessing the destination
- add a delay after accessing the destination
- add a duplicate a request
mikkmokk does fault injection based on a percentage chance. The scope for fault injection may be narrowed further by settings various matching criteria (URI, request method, header name/value pair, etc).
The percentage chances, and related settings, can be set both statically and dynamically:
- Using proxy headers
x-mikkmokk-...when accessing the reverse proxy. - At runtime using the admin API, both for setting new defaults and for introducing one-off errors.
- At startup time using environment variables.
mikkmokk supports proxying to arbitrary URLs.
The following setup proxies to http://example.com:
$ docker run --rm --name mikkmokk-proxy \
-e DESTINATION_URL=http://example.com \
-e PROXY_BIND=0.0.0.0 \
-e PROXY_PORT=8080 \
-e ADMIN_BIND=0.0.0.0 \
-e ADMIN_PORT=7070 \
-p 8080:8080 \
-p 7070:7070 \
docker.io/ivarref/mikkmokk-proxy:v0.1.63There are two ports being exposed:
- The reverse proxy on port 8080.
- The admin server on port 7070.
$ curl http://localhost:8080
...
<h1>Example Domain</h1>
<p>This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.</p>
<p><a href="https://www.iana.org/domains/example">More information...</a></p>
...
This request succeeded because mikkmokk was not told to do any fault injection.
The header x-mikkmokk-fail-before-percentage can be used to simulate that
the destination could not be reached. If present, it must be an int in the range
[0, 100], i.e. it's the percentage chance that a request fails.
The default value for this header is 0.
The value 100 means that the request will always fail.
This can be used to test if clients are retrying or not.
$ curl -v -H 'x-mikkmokk-fail-before-percentage: 100' http://localhost:8080
...
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json
...
{"error":"fail-before"}
The default HTTP status code for this is 503, and may be changed
using the header x-mikkmokk-fail-before-code.
The header x-mikkmokk-fail-after-percentage can be used to simulate that the
destination has received and processed the request, but
that the network between the proxy and the destination failed before
the proxy received the response. Thus, the client will receive an incorrect response.
If the client retries, will the backend handle a duplicate request?
$ curl -v -H 'x-mikkmokk-fail-after-percentage: 100' http://localhost:8080
...
< HTTP/1.1 502 Bad Gateway
< Content-Type: application/json
...
{"error":"fail-after","destination-response-code":200}
The field destination-response-code states which HTTP status code the destination
actually responded with.
The default HTTP status code for this is 502, and may be changed using the header x-mikkmokk-fail-after-code.
The header x-mikkmokk-duplicate-percentage instructs mikkmokk to make two identical, parallel requests.
$ curl -H 'x-mikkmokk-duplicate-percentage: 100' http://localhost:8080
# In the mikkmokk logs you will see something like:
> Duplicate request returned identical HTTP status code 200 for GET http://example.com/
$ curl -H 'x-mikkmokk-match-uri: /something' \
-H 'x-mikkmokk-match-method: GET' \
-H 'x-mikkmokk-fail-before-percentage: 100' \
http://localhost:8080/something
{"error":"fail-before"}
The default value of the x-mikkmokk-match-uri and x-mikkmokk-match-method headers is *, meaning that all URIs and all request methods will match.
If you only want to match a given URI prefix, you may use the x-mikkmokk-match-uri-starts-with header.
$ curl -H 'x-mikkmokk-match-header-name: x-some-header' \
-H 'x-mikkmokk-match-header-value: foobar' \
-H 'x-mikkmokk-fail-before-percentage: 100' \
http://localhost:8080/
... request succeeds, header-name and -value did not match.
$ curl -H 'x-mikkmokk-match-header-name: x-some-header' \
-H 'x-mikkmokk-match-header-value: foobar' \
-H 'x-mikkmokk-fail-before-percentage: 100' \
-H 'x-some-header: foobar' \
http://localhost:8080/
{"error":"fail-before"}
Here we see that the first request did not fail, and thus x-mikkmokk-match-header-name and
x-mikkmokk-header-value did not match.
On the second request it does fail however, and
thus the header name-value pair did match.
We explicitly set x-some-header ourselves.
In a more real world setting it would be set by some gateway.
Delays may be inserted using x-mikkmokk-delay-before-percentage and
x-mikkmokk-delay-before-ms:
$ time curl -H 'x-mikkmokk-delay-before-percentage: 100' \
-H 'x-mikkmokk-delay-before-ms: 3000' \
http://localhost:8080
...
real 0m3.252s
This delay will be inserted before the destination service is accessed.
It's also possible to inject delays after the destination service has
been accessed using x-mikkmokk-delay-after-percentage and
x-mikkmokk-delay-after-ms.
Let's say that you want to test how a frontend handles a failed request, but you do not want edit the source code of the frontend. You also do not want to create any unnecessary errors.
You can use the admin API for one-off errors for these tasks:
# Notice the port 7070 here, which is where we exposed the admin
# API earlier:
$ curl -XPOST -H 'x-mikkmokk-fail-before-percentage: 100' \
http://localhost:7070/api/v1/one-off
{"service":"mikkmokk","message":"Added one-off"}
# The next request now fails:
$ curl http://localhost:8080
{"error":"fail-before"}
# The request after succeeds:
$ curl http://localhost:8080
...<h1>Example Domain</h1>...
The one-off API also supports matching on URI, request method, headers, etc.
The admin API, running on port 7070 in this example, can be used to change the default headers for the runtime of the proxy.
$ curl -XPOST -H 'x-mikkmokk-fail-before-percentage: 20' http://localhost:7070/api/v1/update
{"delay-after-ms":0,
"delay-after-percentage":0,
"delay-before-ms":0,
"delay-before-percentage":0,
"destination-url":"http://example.com",
"duplicate-percentage":0,
"fail-after-code":502,
"fail-after-percentage":0,
"fail-before-code":503,
"fail-before-percentage":20, # <-- fail-before-percentage now has a new default value
"match-header-name":"*",
"match-header-value":"*",
"match-host":"*",
"match-method":"*",
"match-uri":"*",
"match-uri-starts-with":"*"}
# Using the hey load generator https://github.com/rakyll/hey,
# we can test if 20% of requests fail:
$ hey -n 100 http://localhost:8080
...
Status code distribution:
[200] 78 responses
[503] 22 responses
# List current settings
$ curl http://localhost:7070/api/v1/list
{"delay-after-ms":0,
"delay-after-percentage":0,
"delay-before-ms":0,
"delay-before-percentage":0,
"destination-url":"http://example.com",
"duplicate-percentage":0,
"fail-after-code":502,
"fail-after-percentage":0,
"fail-before-code":503,
"fail-before-percentage":20,
"match-header-name":"*",
"match-header-value":"*",
"match-host":"*",
"match-method":"*",
"match-uri":"*",
"match-uri-starts-with":"*"}
# Reset the admin settings
$ curl -XPOST http://localhost:7070/api/v1/reset
{"delay-after-ms":0,
"delay-after-percentage":0,
"delay-before-ms":0,
"delay-before-percentage":0,
"destination-url":"http://example.com",
"duplicate-percentage":0,
"fail-after-code":502,
"fail-after-percentage":0,
"fail-before-code":503,
"fail-before-percentage":0, # <-- fail-before-percentage now has the environment default
"match-header-name":"*",
"match-header-value":"*",
"match-host":"*",
"match-method":"*",
"match-uri":"*",
"match-uri-starts-with":"*"}
$ hey -n 100 http://localhost:8080
...
Status code distribution:
[200] 100 responses
mikkmokk supports a flexible URL proxying scheme. You do not need to create a single mikkmokk instance for every service you want to proxy to. Instead you can tell mikkmokk where to forward to using the URI:
$ curl http://localhost:8080/mikkmokk-forward-http/example.org
... <h1>Example Domain</h1>
# https scheme is also supported
$ curl http://localhost:8080/mikkmokk-forward-https/example.org/some-other-endpoint
... <h1>Example Domain</h1>
mikkmokk will automatically edit the host HTTP header when accessing the destination.
It will also update the origin and Access-Control-Allow-Origin HTTP headers if present.
| Header name | Description | Default value |
|---|---|---|
| delay-after-ms | Number of milliseconds to delay after the destination has replied | 0 |
| delay-after-percentage | Percentage chance of introducing delay after the destination has replied | 0 |
| delay-before-ms | Number of milliseconds to delay before accessing the destination | 0 |
| delay-before-percentage | Percentage chance of introducing delay before accessing the destination | 0 |
| destination-url | Where to forward the request to. E.g. http://example.com | nil |
| duplicate-percentage | Percentage chance of introducing a duplicate request | 0 |
| fail-after-code | The HTTP status code to reply with if a request was deliberately aborted after accessing the destination | 502 |
| fail-after-percentage | Percentage chance of aborting the request after accessing the destination | 0 |
| fail-before-code | The HTTP status code to reply with if a request was deliberately aborted before accessing the destination | 503 |
| fail-before-percentage | Percentage chance of aborting the request before accessing the destination | 0 |
| match-header-name | Only apply failures and/or delays if this HTTP header name's value is identical to ... | * |
| match-header-value | the value in this header. I.e. use this pair of headers to match an arbitrary header value. | * |
| match-host | Only apply failures and/or delays if the destination host matches this value, e.g. example.org |
* |
| match-method | Only apply failures and/or delays to this HTTP method (GET, POST, HEAD, etc.) | * |
| match-uri | Only apply failures and/or delays to this HTTP uri (e.g. /my-api/my-endpoint) |
* |
| match-uri-regex | Only apply failures and/or delays to this HTTP uri if it matches the entire regex. Tip: an uuid may be matched with the following: ([a-f0-9]{8}(-[a-f0-9]{4}){3}-[a-f0-9]{12}) |
* |
| match-uri-starts-with | Only apply failures and/or delays if the HTTP uri starts with this prefix | * |
When using these settings as headers, you will need to prefix them with x-mikkmokk-.
For environment variables, you will need to upper case them and replace dash with underscore, e.g.
destination-url should become DESTINATION_URL.
Should I run mikkmokk-proxy on a public, untrusted network?
No.
Should I run mikkmokk-proxy in production?
No.
Can the logger show the time in my timezone?
Yes. Set the environment property TZ to your timezone, e.g. Europe/Oslo.
NAQ?
Yes, that's Never Asked Questions. ¯\_(ツ)_/¯
No TLS/SSL support for the proxy server (bind). No WebSocket support. No SSE.
There is no attempt at validating -percentage nor -code properties.
-percentage should be [0, 100], and -code should be [200, 600).
envoyproxy has a fault injection filter that seems equivalent to x-mikkmokk-fail-before- headers.
mefellows/muxy: Chaos engineering tool for simulating real-world distributed system failures.
bouncestorage/chaos-http-proxy: Introduce failures into HTTP requests via a proxy server.
clusterfk/chaos-proxy: ClusterFk Chaos Proxy is an unreliable HTTP proxy you can rely on.
toxiproxy: A chaotic TCP proxy.
- Publish multiarch Docker image #1.
- Bump JDK version.
If remote server sends header Access-Control-Allow-Origin in its response,
set it to the input value of Origin.
Remove logger name.
Remove thread name from logging, add information about timezone when logging.
Add coloring of HTTP status codes.
Add x-mikkmokk-match-uri-regex header.
First publicly announced release.
Copyright © 2022 Ivar Refsdal
This program and the accompanying materials are made available under the terms of the Eclipse Public License 2.0 which is available at http://www.eclipse.org/legal/epl-2.0.
This Source Code may also be made available under the following Secondary Licenses when the conditions for such availability set forth in the Eclipse Public License, v. 2.0 are satisfied: GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version, with the GNU Classpath Exception which is available at https://www.gnu.org/software/classpath/license.html.