Skip to content

Commit

Permalink
docs: Gateway Rate Limiting for Cluster Operators
Browse files Browse the repository at this point in the history
  • Loading branch information
@guicassolato
guicassolato committed Aug 28, 2023
1 parent 45afe3d commit 03456eb
Show file tree
Hide file tree
Showing 2 changed files with 134 additions and 174 deletions.
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ The user guides section of the docs gathers several use-cases as well as the ins

* [Simple Rate Limiting for Application Developers](doc/user-guides/simple-rl-for-app-developers.md)
* [Authenticated Rate Limiting for Application Developers](doc/user-guides/authenticated-rl-for-app-developers.md)
* [Gateway rate limiting for cluster operators](doc/user-guides/gateway-rl-for-cluster-operators.md)
* [Gateway Rate Limiting for Cluster Operators](doc/user-guides/gateway-rl-for-cluster-operators.md)
* [Authenticated rate limiting with JWTs and Kubernetes authnz](doc/user-guides/authenticated-rl-with-jwt-and-k8s-authnz.md)

## [Kuadrant Rate Limiting](doc/rate-limiting.md)
Expand Down
306 changes: 133 additions & 173 deletions doc/user-guides/gateway-rl-for-cluster-operators.md
View file
Original file line number Diff line number Diff line change
@@ -1,244 +1,204 @@
# Gateway Rate Limit For Cluster Operators
# Gateway Rate Limiting for Cluster Operators

This user guide shows how the kuadrant's control plane applies rate limit policy at
[Gateway API's Gateway](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1beta1.Gateway)
level.
This user guide walks you through an example of how to configure rate limiting for all routes attached to an ingress gateway.

### Clone the project
<br/>

## Run the steps ① → ④

### ① Setup

This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using [Kind](https://kind.sigs.k8s.io),
where it installs Istio, Kubernetes Gateway API and Kuadrant itself.

> **Note:** In production environment, these steps are usually performed by a cluster operator with administrator privileges over the Kubernetes cluster.
Clone the project:

```sh
git clone https://github.com/Kuadrant/kuadrant-operator && cd kuadrant-operator
```

### Setup environment

This step creates a containerized Kubernetes server locally using [Kind](https://kind.sigs.k8s.io),
then it installs Istio, Kubernetes Gateway API and kuadrant.
Setup the environment:

```sh
make local-setup
```

### Apply Kuadrant CR
Request an instance of Kuadrant:

```sh
kubectl -n kuadrant-system apply -f - <<EOF
---
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
name: kuadrant-sample
name: kuadrant
spec: {}
EOF
```

### Deploy toystore example deployment
### ② Create the ingress gateways

```sh
kubectl apply -f examples/toystore/toystore.yaml
```

### Create HTTPRoute to configure routing to the toystore service

![](https://i.imgur.com/rdN8lo3.png)

```sh
kubectl apply -f - <<EOF
kubectl -n istio-system apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: external
annotations:
kuadrant.io/namespace: kuadrant-system
networking.istio.io/service-type: ClusterIP
spec:
gatewayClassName: istio
listeners:
- name: external
port: 80
protocol: HTTP
hostname: '*.io'
allowedRoutes:
namespaces:
from: All
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
kind: Gateway
metadata:
name: toystore
labels:
app: toystore
name: internal
annotations:
kuadrant.io/namespace: kuadrant-system
networking.istio.io/service-type: ClusterIP
spec:
parentRefs:
- name: istio-ingressgateway
namespace: istio-system
hostnames: ["*.toystore.com"]
rules:
- matches:
- path:
type: PathPrefix
value: "/toy"
method: GET
- path:
type: PathPrefix
value: "/free"
method: GET
- path:
type: Exact
value: "/admin/toy"
method: POST
backendRefs:
- name: toystore
port: 80
gatewayClassName: istio
listeners:
- name: local
port: 80
protocol: HTTP
hostname: '*.local'
allowedRoutes:
namespaces:
from: All
EOF
```

### Check `toystore` HTTPRoute works

`GET /toy`: no rate limiting
```sh
while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Host: api.toystore.com" http://localhost:9080/toy | egrep --color "\b(429)\b|$"; sleep 1; done
```
### ③ Enforce rate limiting on requests incoming through the `external` gateway

`POST /admin/toy`: no rate limiting
```sh
while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Host: api.toystore.com" -X POST http://localhost:9080/admin/toy | egrep --color "\b(429)\b|$"; sleep 1; done
```

**Note**: This only works out of the box on linux environments. If not on linux,
you may need to forward ports

```bash
kubectl port-forward -n istio-system service/istio-ingressgateway 9080:80
┌───────────┐ ┌───────────┐
│ (Gateway) │ │ (Gateway) │
│ external │ │ internal │
│ │ │ │
│ *.io │ │ *.local │
└───────────┘ └───────────┘
▲
│
┌─────────┴─────────┐
│ (RateLimitPolicy) │
│ gw-rlp │
└───────────────────┘
```

### Rate limiting `toystore` HTTPRoute traffic

![](https://i.imgur.com/2A9sXXs.png)

RateLimitPolicy applied for the `toystore` HTTPRoute.

| Endpoints | Rate Limits |
|-------------------|------------------------------------:|
| `POST /admin/toy` | **5** reqs / **10** secs (0.5 rps) |
| `GET /toy` | **8** reqs / **10** secs (0.8 rps) |
| `*` | **30** reqs / **10** secs (3.0 rps) |
Create a Kuadrant `RateLimitPolicy` to configure rate limiting:

```sh
kubectl apply -f - <<EOF
---
apiVersion: kuadrant.io/v1beta1
kubectl apply -n istio-system -f - <<EOF
apiVersion: kuadrant.io/v1beta2
kind: RateLimitPolicy
metadata:
name: toystore
name: gw-rlp
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: toystore
rateLimits:
- rules:
- paths: ["/admin/toy"]
methods: ["POST"]
configurations:
- actions:
- generic_key:
descriptor_key: admin_operation
descriptor_value: "1"
limits:
- conditions:
- "admin_operation == '1'"
maxValue: 5
seconds: 10
variables: []
- rules:
- paths: ["/toy"]
methods: ["GET"]
configurations:
- actions:
- generic_key:
descriptor_key: get_operation
descriptor_value: "1"
limits:
- conditions:
- "get_operation == '1'"
maxValue: 8
seconds: 10
variables: []
- configurations:
- actions:
- generic_key:
descriptor_key: toystore
descriptor_value: "1"
limits:
- conditions: ["toystore == '1'"]
maxValue: 30
seconds: 10
variables: []
kind: Gateway
name: external
limits:
"global":
rates:
- limit: 5
duration: 10
unit: second
EOF
```

Validating the rate limit policy.
> **Note:** It may take a couple of minutes for the RateLimitPolicy to be applied depending on your cluster.
`GET /toy` @ **1** rps (expected to be rate limited @ **8** reqs / **10** secs (0.8 rps))
```sh
while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Host: api.toystore.com" http://localhost:9080/toy | egrep --color "\b(429)\b|$"; sleep 1; done
```
### ④ Deploy a sample API to test rate limiting enforced at the level of the gateway

`POST /admin/toy` @ **1** rps (expected to be rate limited @ **5** reqs / **10** secs (0.5 rps))
```sh
while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Host: api.toystore.com" -X POST http://localhost:9080/admin/toy | egrep --color "\b(429)\b|$"; sleep 1; done
```
┌───────────┐ ┌───────────┐
┌───────────────────┐ │ (Gateway) │ │ (Gateway) │
│ (RateLimitPolicy) │ │ external │ │ internal │
│ gw-rlp ├─────►│ │ │ │
└───────────────────┘ │ *.io │ │ *.local │
└─────┬─────┘ └─────┬─────┘
│ │
└─────────┬────────┘
│
┌─────────┴────────┐
│ (HTTPRoute) │
│ toystore │
│ │
│ *.toystore.io │
│ *.toystore.local │
└────────┬─────────┘
│
┌──────┴───────┐
│ (Service) │
│ toystore │
└──────────────┘
```

### Rate limiting Gateway traffic

![](https://i.imgur.com/0o3yQzP.png)
Deploy the sample API:

RateLimitPolicy applied for the Gateway.
```sh
kubectl apply -f examples/toystore/toystore.yaml
```

| Policy | Rate Limits |
|---------------|------------------------------------:|
| `POST /*` | **2** reqs / **10** secs (0.2 rps) |
| Per remote IP | **25** reqs / **10** secs (2.5 rps) |
Route traffic to the API from both gateways:

```sh
kubectl apply -f - <<EOF
---
apiVersion: kuadrant.io/v1beta1
kind: RateLimitPolicy
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: kuadrant-gw
namespace: istio-system
name: toystore
spec:
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: istio-ingressgateway
rateLimits:
- rules:
- methods: ["POST"]
configurations:
- actions:
- generic_key:
descriptor_key: expensive_op
descriptor_value: "1"
limits:
- conditions: ["expensive_op == '1'"]
maxValue: 2
seconds: 10
variables: []
- configurations:
- actions:
- remote_address: {}
limits:
- conditions: []
maxValue: 25
seconds: 10
variables: ["remote_address"]
parentRefs:
- name: external
namespace: istio-system
- name: internal
namespace: istio-system
hostnames:
- "*.toystore.io"
- "*.toystore.local"
rules:
- backendRefs:
- name: toystore
port: 80
EOF
```

### Validating the rate limit policies (HTTPRoute and Gateway).
### ⑤ Verify the rate limiting works by sending requests in a loop

Expose the gateways, respectively at the port numbers `9080` and `9081` of the local host:

`GET /toy` @ **1** rps (expected to be rate limited @ **8** reqs / **10** secs (0.8 rps))
```sh
while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Host: api.toystore.com" http://localhost:9080/toy | egrep --color "\b(429)\b|$"; sleep 1; done
kubectl port-forward -n istio-system service/external-istio 9080:80 2>&1 >/dev/null &
kubectl port-forward -n istio-system service/internal-istio 9081:80 2>&1 >/dev/null &
```

`POST /admin/toy` @ **1** rps (expected to be rate limited @ **2** reqs / **10** secs (0.2 rps))
Up to 5 successful (`200 OK`) requests every 10 seconds through the `external` ingress gateway (`*.io`), then `429 Too Many Requests`:

```sh
while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Host: api.toystore.com" -X POST http://localhost:9080/admin/toy | egrep --color "\b(429)\b|$"; sleep 1; done
while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H 'Host: api.toystore.io' http://localhost:9080 | egrep --color "\b(429)\b|$"; sleep 1; done
```

### Validating Gateway "Per Remote IP" policy
Unlimited successful (`200 OK`) through the `internal` ingress gateway (`*.local`):

Stop all traffic.
```sh
while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H 'Host: api.toystore.local' http://localhost:9081 | egrep --color "\b(429)\b|$"; sleep 1; done
```

`GET /free` @ **3** rps (expected to be rate limited @ **25** reqs / **10** secs (2.5 rps))
## Cleanup

```sh
while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null -H "Host: api.toystore.com" http://localhost:9080/free -: --write-out '%{http_code}\n' --silent --output /dev/null -H "Host: api.toystore.com" http://localhost:9080/free -: --write-out '%{http_code}\n' --silent --output /dev/null -H "Host: api.toystore.com" http://localhost:9080/free | egrep --color "\b(429)\b|$"; sleep 1; done
make local-cleanup
```

0 comments on commit 03456eb

Please sign in to comment.