diff --git a/README.md b/README.md index 1a216a5e0..743a7491f 100644 --- a/README.md +++ b/README.md @@ -161,10 +161,10 @@ EOF The user guides section of the docs gathers several use-cases as well as the instructions to implement them using kuadrant. -* [Simple rate limiting for API owners](doc/user-guides/simple-rl-for-api-owners.md) -* [Authenticated rate limiting for API owners](doc/user-guides/authenticated-rl-for-api-owners.md) -* [Gateway rate limiting for cluster operators](doc/user-guides/gateway-rl-for-cluster-operators.md) -* [Authenticated rate limiting with JWTs and Kubernetes authnz](doc/user-guides/authenticated-rl-with-jwt-and-k8s-authnz.md) +* [Simple Rate Limiting for Application Developers](doc/user-guides/simple-rl-for-app-developers.md) +* [Authenticated Rate Limiting for Application Developers](doc/user-guides/authenticated-rl-for-app-developers.md) +* [Gateway Rate Limiting for Cluster Operators](doc/user-guides/gateway-rl-for-cluster-operators.md) +* [Authenticated Rate Limiting with JWTs and Kubernetes RBAC](doc/user-guides/authenticated-rl-with-jwt-and-k8s-authnz.md) ## [Kuadrant Rate Limiting](doc/rate-limiting.md) diff --git a/doc/rate-limiting.md b/doc/rate-limiting.md index 251037431..6cb90b689 100644 --- a/doc/rate-limiting.md +++ b/doc/rate-limiting.md @@ -1,85 +1,99 @@ # Kuadrant Rate Limiting -## Goals +A Kuadrant RateLimitPolicy custom resource, often abbreviated "RLP": -Kuadrant sees the following requirements for an **ingress gateway** based rate limit policy: - -* Allow it to target **routing/network** resources such as -[HTTPRoute](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPRoute) -and [Gateway](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1beta1.Gateway) and use these resources to provide needed context (which traffic workload (hostname), which gateway). -* Use it to define when to invoke rate limiting (what paths, what methods etc) and the needed -metadata IE actions and descriptors that are needed to enforce the rate limiting requirements. -* Avoid exposing the end user to the complexity of the underlying configuration resources that has -a much broader remit and surface area. -* Allow administrators (cluster operators) to set overrides and defaults that govern what can be -done at the lower levels. +1. Allows it to target Gateway API networking resources such as [HTTPRoutes](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPRoute) and [Gateways](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.Gateway), using these resources to obtain additional context, i.e., which traffic workload (HTTP attributes, hostnames, user attributes, etc) to rate limit. +2. Allows to specify which specific subsets of the targeted network resource to apply the limits to. +3. Abstracts the details of the underlying Rate Limit protocol and configuration resources, that have a much broader remit and surface area. +4. Supports cluster operators to set overrides (soon) and defaults that govern what can be done at the lower levels. ## How it works -### Envoy's Rate Limit Service Potocol +### Envoy's Rate Limit Service Protocol + +Kuadrant's Rate Limit implementation relies on the Envoy's [Rate Limit Service (RLS)](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ratelimit/v3/rls.proto) protocol. The workflow per request goes: +1. On incoming request, the gateway checks the matching rules for enforcing rate limits, as stated in the RateLimitPolicy custom resources and targeted Gateway API networking objects +2. If the request matches, the gateway sends one [RateLimitRequest](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ratelimit/v3/rls.proto#service-ratelimit-v3-ratelimitrequest) to the external rate limiting service ("Limitador"). +1. The external rate limiting service responds with a [RateLimitResponse](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ratelimit/v3/rls.proto#service-ratelimit-v3-ratelimitresponse) back to the gateway with either an `OK` or `OVER_LIMIT` response code. + +A RateLimitPolicy and its targeted Gateway API networking resource contain all the statements to configure both the ingress gateway and the external rate limiting service. -Kuadrant's rate limit implementation relies on the -[Rate Limit Service (RLS)](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ratelimit/v3/rls.proto) -protocol. The workflow per request would be: +### The RateLimitPolicy custom resource -1. On incoming request, the gateway sends (optionally, depending on the context) -one [RateLimitRequest](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ratelimit/v3/rls.proto#service-ratelimit-v3-ratelimitrequest) -to the external rate limiting service. -2. The external rate limiting service answers with a [RateLimitResponse](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ratelimit/v3/rls.proto#service-ratelimit-v3-ratelimitresponse) -back to the gateway with either `OK` or `OVER_LIMIT` response code. +#### Overview -The RateLimitPolicy contains the bits to configure both the gateway and the external rate limiting service. +The `RateLimitPolicy` spec includes, basically, two parts: -### The RateLimitPolicy object overview +* A reference to an existing Gateway API resource (`spec.targetRef`) +* Limit definitions (`spec.limits`) -The `RateLimitPolicy` resource includes, basically, three parts: +Each limit definition includes: +* A set of rate limits (`spec.limits..rates[]`) +* (Optional) A set of dynamic counter qualifiers (`spec.limits..counters[]`) +* (Optional) A set of route selectors, to further qualify the specific routing rules when to activate the limit (`spec.limits..routeSelectors[]`) +* (Optional) A set of additional dynamic conditions to activate the limit (`spec.limits..when[]`) -* A reference to existing routing/networing Gateway API resource. - * location: `spec.targetRef` -* Gateway configuration to produce rate limit descriptors. - * location: `spec.rateLimits[].configurations` and `spec.rateLimits[].rules` -* External rate limiting service, [Limitador's](https://github.com/Kuadrant/limitador) configuration. - * location: `spec.rateLimits[].limits` + + + + + + +
Check out Kuadrant RFC 0002 to learn more about the Well-known Attributes that can be used to define counter qualifiers (counters) and conditions (when).
+ +#### High-level example and field definition ```yaml ---- -apiVersion: kuadrant.io/v1beta1 +apiVersion: kuadrant.io/v1beta2 kind: RateLimitPolicy metadata: name: my-rate-limit-policy spec: - # targetRef defines a reference to existing routing/networking resource object to apply policy to. - targetRef: { ... } + # reference to an existing networking resource to attach the policy to + # it can be a Gateway API HTTPRoute or Gateway resource + # it can only refer to objects in the same namespace as the RateLimitPolicy + targetRef: group: gateway.networking.k8s.io kind: HTTPRoute / Gateway name: myroute / mygateway - rateLimits: - # Rules defines the list of conditions for which rate limit configuration will apply. - # Used to configure ingress gateway. - - rules: [ ... ] - # Each configuration object represents one action configuration. - # Each configuration produces, at most, one rate limit descriptor. - # Used to configure ingress gateway. - configurations: [ ... ] - # Limits are used to configure rate limiting boundaries on time periods. - # Used to configure kuadrant's external rate limiting service. - limits: [ ... ] + + # the limits definitions to apply to the network traffic routed through the targeted resource + limits: + "my_limit": + # the rate limits associated with this limit definition + # e.g., to specify a 50rps rate limit, add `{ limit: 50, duration: 1, unit: secod }` + rates: […] + + # (optional) counter qualifiers + # each dynamic value in the data plane starts a separate counter, combined with each rate limit + # e.g., to define a separate rate limit for each user name detected by the auth layer, add `metadata.filter_metadata.envoy\.filters\.http\.ext_authz.username` + # check out Kuadrant RFC 0002 (https://github.com/Kuadrant/architecture/blob/main/rfcs/0002-well-known-attributes.md) to learn more about the Well-known Attributes that can be used in this field + counters: […] + + # (optional) further qualification of the scpecific HTTPRouteRules within the targeted HTTPRoute that should trigger the limit + # each element contains a HTTPRouteMatch object that will be used to select HTTPRouteRules that include at least one identical HTTPRouteMatch + # the HTTPRouteMatch part does not have to be fully identical, but the what's stated in the selector must be identically stated in the HTTPRouteRule + # do not use it on RateLimitPolicies that target a Gateway + routeSelectors: […] + + # (optional) additional dynamic conditions to trigger the limit. + # use it for filterring attributes not supported by HTTPRouteRule or with RateLimitPolicies that target a Gateway + # check out Kuadrant RFC 0002 (https://github.com/Kuadrant/architecture/blob/main/rfcs/0002-well-known-attributes.md) to learn more about the Well-known Attributes that can be used in this field + when: […] ``` ## Using the RateLimitPolicy ### Targeting a HTTPRoute networking resource -When a rate limit policy targets an HTTPRoute, the policy is scoped by the domains defined -at the referenced HTTPRoute's hostnames. +When a RLP targets a HTTPRoute, the policy is enforced to all traffic routed according to the rules and hostnames specified in the HTTPRoute, across all Gateways referenced in the `spec.parentRefs` field of the HTTPRoute. -The rate limit policy targeting an HTTPRoute will be applied to every single ingress gateway -referenced by the HTTPRoute in the `spec.parentRefs` field. +The targeted HTTPRoute's rules and/or hostnames to which the policy must be enforced can be filtered to specific subsets, by specifying the `routeSelectors` field of the limit definition. -Targeting is defined with the `spec.targetRef` field, as follows: +Target a HTTPRoute by setting the `spec.targetRef` field of the RLP as follows: ```yaml -apiVersion: kuadrant.io/v1beta1 +apiVersion: kuadrant.io/v1beta2 kind: RateLimitPolicy metadata: name: @@ -88,56 +102,43 @@ spec: group: gateway.networking.k8s.io kind: HTTPRoute name: - rateLimits: [ ... ] + limits: {…} ``` -![](https://i.imgur.com/ObfOp9u.png) +![Rate limit policy targeting a HTTPRoute resource](https://i.imgur.com/ObfOp9u.png) -**Multiple HTTPRoutes with the same hostname** +#### Multiple HTTPRoutes with the same hostname -When there are multiple HTTPRoutes with the same hostname, -HTTPRoutes are all admitted and the ingress gateway will merge the routing configurations -in the same virtualhost. In these cases, kuadrant control plane will also merge rate limit -policies referencing HTTPRoutes with the same hostname. +When multiple HTTPRoutes state the same hostname, these HTTPRoutes are usually all admitted and merged together by the gateway implemetation in the same virtual host configuration of the gateway. Similarly, the Kuadrant control plane will also register all rate limit policies referencing the HTTPRoutes, activating the correct limits across policies according to the routing matching rules of the targeted HTTPRoutes. -**Overlapping HTTPRoutes** +#### Hostnames and wildcards -If one RLP targets a route for `*.com` and other RLP targets another route for `api.com`, -the kuadrant's control plane does not do any *merging*. A request coming for `api.com` will be -rate limited according to the rules from the RLP targeting the route `api.com`. -On the other hand, a request coming for `other.com` will be rate limited with the rules -from the RLP targeting the route `*.com`. - -For example, let's say we have three rate limit policies in place: - -``` -RLP A -> HTTPRoute A (api.toystore.com) - -RLP B -> HTTPRoute B (other.toystore.com) - -RLP H -> HTTPRoute H (*.toystore.com) -``` +If a RLP targets a route defined for `*.com` and another RLP targets another route for `api.com`, the Kuadrant control plane will not merge these two RLPs. Rather, it will mimic the behavior of gateway implementation by which the "most specific hostname wins", thus enforcing only the corresponding applicable policies and limit definitions. -Request 1 (api.toystore.com) -> RLP A will be applied +E.g., a request coming for `api.com` will be rate limited according to the rules from the RLP that targets the route for `api.com`; while a request for `other.com` will be rate limited with the rules from the RLP targeting the route for `*.com`. -Request 2 (other.toystore.com) -> RLP B will be applied +Example with 3 RLPs and 3 HTTPRoutes: +- RLP A → HTTPRoute A (`a.toystore.com`) +- RLP B → HTTPRoute B (`b.toystore.com`) +- RLP W → HTTPRoute W (`*.toystore.com`) -Request 3 (unknown.toystore.com) -> RLP H will be applied +Expected behavior: +- Request to `a.toystore.com` → RLP A will be enforced +- Request to `b.toystore.com` → RLP B will be enforced +- Request to `other.toystore.com` → RLP W will be enforced ### Targeting a Gateway networking resource -A key use case is being able to provide governance over what service providers can and cannot do -when exposing a service via a shared ingress gateway. As well as providing certainty that -no service is exposed without my ability as a cluster administrator to protect my infrastructure -from unplanned load from badly behaving clients etc. +When a RLP targets a Gateway, the policy will be enforced to all HTTP traffic hitting the gateway, unless a more specific RLP targeting a matching HTTPRoute exists. -When a rate limit policy targets Gateway, the policy will be applied to all HTTP traffic hitting -the gateway. +Any new HTTPRoute referrencing the gateway as parent will be automatically covered by the RLP that targets the Gateway, as well as changes in the existing HTTPRoutes. -Targeting is defined with the `spec.targetRef` field, as follows: +This effectively provides cluster operators with the ability to set _defaults_ to protect the infrastructure against unplanned and malicious network traffic attempt, such as by setting preemptive limits for hostnames and hostname wildcards. + +Target a Gateway HTTPRoute by setting the `spec.targetRef` field of the RLP as follows: ```yaml -apiVersion: kuadrant.io/v1beta1 +apiVersion: kuadrant.io/v1beta2 kind: RateLimitPolicy metadata: name: @@ -146,243 +147,250 @@ spec: group: gateway.networking.k8s.io kind: Gateway name: - rateLimits: [ ... ] -``` - -![](https://i.imgur.com/UkivAqA.png) - -The kuadrant control plane will aggregate all the rate limit policies that apply to a gateway, -including multiple RLP targeting HTTPRoutes and Gateways. For example, -let's say we have four rate limit policies in place: - + limits: {…} ``` -RLP A -> HTTPRoute A (`api.toystore.com`) -> Gateway G (`*.com`) -RLP B -> HTTPRoute B (`other.toystore.com`) -> Gateway G (`*.com`) +![rate limit policy targeting a Gateway resource](https://i.imgur.com/UkivAqA.png) -RLP H -> HTTPRoute H (`*.toystore.com`) -> Gateway G (`*.com`) +#### Overlapping Gateway and HTTPRoute RLPs -RLP G -> Gateway G (`*.com`) -``` +Gateway-targeted RLPs will serve as a default to protect all traffic routed through the gateway until a more specific HTTPRoute-targeted RLP exists, in which case the HTTPRoute RLP prevails. -Request 1 (`api.toystore.com`) -> apply RLP A and RLP G +Example with 4 RLPs, 3 HTTPRoutes and 1 Gateway (plus 2 HTTPRoute and 2 Gateways without RLPs attached): +- RLP A → HTTPRoute A (`a.toystore.com`) → Gateway G (`*.com`) +- RLP B → HTTPRoute B (`b.toystore.com`) → Gateway G (`*.com`) +- RLP W → HTTPRoute W (`*.toystore.com`) → Gateway G (`*.com`) +- RLP G → Gateway G (`*.com`) -Request 2 (`other.toystore.com`) -> apply RLP B and RLP G +Expected behavior: +- Request to `a.toystore.com` → RLP A will be enforced +- Request to `b.toystore.com` → RLP B will be enforced +- Request to `other.toystore.com` → RLP W will be enforced +- Request to `other.com` (suppose a route exists) → RLP G will be enforced +- Request to `yet-another.net` (suppose a route and gateway exist) → No RLP will be enforced -Request 3 (`unknown.toystore.com`) -> apply RLP H and RLP G +### Limit definition -Request 4 (`other.com`) -> apply RLP G +A limit will be activated whenever a request comes in and the request matches: +- any of the route rules selected by the limit (via `routeSelectors` or implicit "catch-all" selector), and +- all of the `when` conditions specified in the limit. -**Note**: When a request falls under the scope of multiple policies, all the policies will be applied. -Following the rate limiting design guidelines, the most restrictive policy will be enforced. +A limit can define: +- counters that are qualified based on dynamic values fetched from the request, or +- global counters (implicitly, when no qualified counter is specified) -### Action configurations +A limit is composed of one or more rate limits. -Action configurations are defined via rate limit configuration objects. -The rate limit configuration object is the equivalent of the -[config.route.v3.RateLimit](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#envoy-v3-api-msg-config-route-v3-ratelimit) envoy object. -One configuration is, in turn, a list of -[rate limit actions](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#envoy-v3-api-msg-config-route-v3-ratelimit-action). -Each action populates a descriptor entry. A list of descriptor entries compose a descriptor. -A list of descriptors compose a [RateLimitRequest](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ratelimit/v3/rls.proto#service-ratelimit-v3-ratelimitrequest). -Each configuration produces, at most, one descriptor. -Depending on the incoming request, one configuration may or may not produce a rate limit descriptor. -These rate limiting configuration rules provide flexibility to produce multiple descriptors. - -An example to illustrate +E.g. ```yaml -configurations: - - actions: - - request_headers: - header_name: "X-MY-CUSTOM-HEADER" - descriptor_key: "custom-header" - skip_if_absent: true - - actions: - - generic_key: - descriptor_key: admin - descriptor_value: "1" +spec: + limits: + "toystore-all": + rates: + - limit: 5000 + duration: 1 + unit: second + + "toystore-api-per-username": + rates: + - limit: 100 + duration: 1 + unit: second + - limit: 1000 + duration: 1 + unit: minute + counters: + - auth.identity.username + routeSelectors: + hostnames: + - api.toystore.com + + "toystore-admin-unverified-users": + rates: + - limit: 250 + duration: 1 + unit: second + routeSelectors: + hostnames: + - admin.toystore.com + when: + - selector: auth.identity.email_verified + operator: eq + value: "false" ``` -A request without "X-MY-CUSTOM-HEADER" would generate one descriptor with one entry: +| Request to | Rate limits enforced | +|----------------------|--------------------------------------------------------------| +| `api.toystore.com` | 100rps/username or 1000rpm/username (whatever happens first) | +| `admin.toystore.com` | 250rps | +| `other.toystore.com` | 5000rps | -``` -("admin": "1") -``` +#### Route selectors -A request with a header "X-MY-CUSTOM-HEADER=MY-VALUE" would generate two descriptors, -one entry each descriptor: +The `routeSelectors` field of the limit definition allows to specify **selectors of routes** (or parts of a route), that _transitively induce a set of conditions for a limit to be enforced_. It is defined as a set of route matching rules, where these rules must exist, partially or identically stated within the HTTPRouteRules of the HTTPRoute that is targeted by the RLP. -``` -("admin": "1") -("custom-header": "MY-VALUE") -``` +The field is typed as a list of objects based on a special type defined from Gateway API's [HTTPRouteMatch](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPPathMatch) type (`matches` subfield of the route selector object), and an additional field `hostnames`. -**Note**: If one action is not able to populate a descriptor entry, the entire descriptor is discarded. +Route selectors matches and the HTTPRoute's HTTPRouteMatches are pairwise compared to select or not select HTTPRouteRules that should activate a limit. To decide whether the route selector selects a HTTPRouteRule or not, for each pair of route selector HTTPRouteMatch and HTTPRoute HTTPRouteMatch: +1. The route selector selects the HTTPRoute's HTTPRouteRule if the HTTPRouteRule contains at least one HTTPRouteMatch that specifies fields that are literally identical to all the fields specified by at least one HTTPRouteMatch of the route selector. +2. A HTTPRouteMatch within a HTTPRouteRule may include other fields that are not specified in a route selector match, and yet the route selector match selects the HTTPRouteRule if all fields of the route selector match are identically included in the HTTPRouteRule's HTTPRouteMatch; the opposite is NOT true. +3. Each field `path` of a HTTPRouteMatch, as well as each field `method` of a HTTPRouteMatch, as well as each element of the fields `headers` and `queryParams` of a HTTPRouteMatch, is atomic – this is true for the HTTPRouteMatches within a HTTPRouteRule, as well as for HTTPRouteMatches of a route selector. -**Note**: The external rate limiting service will be called only when there is at least one not empty -descriptor. +Additionally, at least one hostname specified in a route selector must identically match one of the hostnames specified (or inherited, when omitted) by the targeted HTTPRoute. -### Policy default action configurations +The semantics of the route selectors allows to assertively relate limit definitions to routing rules, with benefits for identifying the subsets of the network that are covered by a limit, while preventing unreachable definitions, as well as the overhead associated with the maintenance of such rules across multiple resources throughout time, according to network topology beneath. Moreover, the requirement of not having to be a full copy of the targeted HTTPRouteRule matches, but only partially identical, helps prevent repetition to some degree, as well as it enables to more easily define limits that scope across multiple HTTPRouteRules (by specifying less rules in the selector). -When a rate limit policy does not specify any action configuration, the Kuadrant control plane -will assign a generic default action configuration for the traffic related to the targeted network -resource. This default action configuration allows defining global limits for all the traffic -related the targeted network resource. For instance, the following rate limit policy is valid: +A few rules and corner cases to keep in mind while using the RLP's `routeSelectors`: +1. **The golden rule –** The route selectors in a RLP are **not** to be read strictly as the route matching rules that activate a limit, but as selectors of the route rules that activate the limit. +2. Due to (1) above, this can lead to cases, e.g., where a route selector that states `matches: [{ method: POST }]` selects a HTTPRouteRule that defines `matches: [{ method: POST }, { method: GET }]`, effectively causing the limit to be activated on requests to the HTTP method `POST`, but **also** to the HTTP method `GET`. +3. The requirement for the route selector match to state patterns that are identical to the patterns stated by the HTTPRouteRule (partially or entirely) makes, e.g., a route selector such as `matches: { path: { type: PathPrefix, value: /foo } }` to select a HTTPRouteRule that defines `matches: { path: { type: PathPrefix, value: /foo }, method: GET }`, but **not** to select a HTTPRouteRule that only defines `matches: { method: GET }`, even though the latter includes technically all HTTP paths; **nor** it selects a HTTPRouteRule that only defines `matches: { path: { type: Exact, value: /foo } }`, even though all requests to the exact path `/foo` are also technically requests to `/foo*`. +4. The atomicity property of fields of the route selectors makes, e.g., a route selector such as `matches: { path: { value: /foo } }` to select a HTTPRouteRule that defines `matches: { path: { value: /foo } }`, but **not** to select a HTTPRouteRule that only defines `matches: { path: { type: PathPrefix, value: /foo } }`. (This case may actually never happen because `PathPrefix` is the default value for `path.type` and will be set automatically by the Kubernetes API server.) -```yaml -spec: - targetRef: { ... } - rateLimits: - - limits: - - maxValue: 5 - seconds: 10 -``` +Due to the nature of route selectors of defining pointers to HTTPRouteRules, the `routeSelectors` field is not supported in a RLP that targets a Gateway resource. -### Rate limiting configuration rules +#### `when` conditions -Configuration rules allow rate limit configurations to be activated conditionally depending on -the current context (the incoming HTTP request properties). -Each rate limit configuration list can define, optionally, a list of rules to match the request. -A match occurs when *at least* one rule matches the request. +`when` conditions can be used to scope a limit (i.e. to filter the traffic to which a limit definition applies) without any coupling to the underlying network topology, i.e. without making direct references to HTTPRouteRules via `routeSelectors`. -An example to illustrate. Given these rate limit configurations, +The syntax of the `when` conditions selectors comply with Kuadrant's [Well-known Attributes (RFC 0002)](https://github.com/Kuadrant/architecture/blob/main/rfcs/0002-well-known-attributes.md). -```yaml -spec: - rateLimits: - - configurations: - - actions: - - generic_key: - descriptor_key: toystore-app - descriptor_value: "1" - - rules: - - hosts: ["api.toystore.com"] - configurations: - - actions: - - generic_key: - descriptor_key: api - descriptor_value: "1" - - rules: - - hosts: ["admin.toystore.com"] - configurations: - - actions: - - generic_key: - descriptor_key: admin - descriptor_value: "1" -``` +Use the `when` conditions to conditionally activate limits based on attributes that cannot be expressed in the HTTPRoutes' `spec.hostnames` and `spec.rules.matches` fields, or in general in RLPs that target a Gateway. -* When a request for `api.toystore.com` hits the gateway, the descriptors generated would be: +### Examples -``` -("toystore-app", "1") -("api", "1") -``` +Check out the following user guides for examples of rate limiting services with Kuadrant: +* [Simple Rate Limiting for Application Developers](user-guides/simple-rl-for-app-developers.md) +* [Authenticated Rate Limiting for Application Developers](user-guides/authenticated-rl-for-app-developers.md) +* [Gateway Rate Limiting for Cluster Operators](user-guides/gateway-rl-for-cluster-operators.md) +* [Authenticated Rate Limiting with JWTs and Kubernetes RBAC](user-guides/authenticated-rl-with-jwt-and-k8s-authnz.md) -* When a request for `admin.toystore.com` hits the gateway, the descriptors generated would be: +### Known limitations -``` -("toystore-app", "1") -("admin", "1") -``` +* One HTTPRoute can only be targeted by one RLP. +* One Gateway can only be targeted by one RLP. +* RLPs can only target HTTPRoutes/Gateways defined within the same namespace of the RLP. -* When a request for `other.toystore.com` hits the gateway, the descriptors generated would be: +## Implementation details -``` -("toystore-app", "1") -``` +Driven by limitations related to how Istio injects configuration in the filter chains of the ingress gateways, Kuadrant relies on Envoy's [Wasm Network](https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/wasm_filter) filter in the data plane, to manage the integration with rate limiting service ("Limitador"), instead of the [Rate Limit](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/rate_limit_filter) filter. -**Note**: If rules are not set, it is equivalent to matching all the requests. +**Motivation:** _Multiple rate limit domains_
+The first limitation comes from having only one filter chain per listener. This often leads to one single global rate limiting filter configuration per gateway, and therefore to a shared rate limit [domain](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ratelimit/v3/rate_limit.proto#envoy-v3-api-msg-extensions-filters-http-ratelimit-v3-ratelimit) across applications and policies. Even though, in a rate limit filter, the triggering of rate limit calls, via [actions to build so-called "descriptors"](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/rate_limit_filter#composing-actions), can be defined at the level of the virtual host and/or specific route rule, the overall rate limit configuration is only one, i.e., always the same rate limit domain for all calls to Limitador. -### Known limitations +On the other hand, the possibility to configure and invoke the rate limit service for multiple domains depending on the context allows to isolate groups of policy rules, as well as to optimize performance in the rate limit service, which can rely on the domain for indexation. -* One HTTPRoute can only be targeted by one rate limit policy. -* One Gateway can only be targeted by one rate limit policy. -* Only supporting HTTPRoute/Gateway references from within the same namespace. -* `hosts` in rules, `spec.rateLimits[].rules`, do not support wildcard prefixes. - -## How: Implementation details - -### The WASM Filter - -On designing kuadrant rate limiting and considering Istio/Envoy's rate limiting offering, -we hit two limitations. - -* *Shared Rate Limiting Domain*: The rate limiting -[domain](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ratelimit/v3/rate_limit.proto#envoy-v3-api-msg-extensions-filters-http-ratelimit-v3-ratelimit) -used in the global rate limiting filter in Envoy are shared across the Ingress Gateway. -This is because Istio creates only one filter chain by default at the listener level. -This means the rate limiting filter configuration is shared at the gateway level -(which rate limiting service to call, which domain to use). The triggering of actual rate limiting -calls happens at the -[virtual host / route level by adding actions and descriptors](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/rate_limit_filter#rate-limit). -This need to have shared domains causes several issues: - * All rate limit configurations applied to limitador need to use a shared domain or set of - shared domains (when using stages). This means that for each rate limiting request, - limitador will need to iterate through each of the rate limit resources within the shared domain - and evaluate each of their conditions to find which one applies. As the number of APIs increases - so would the number of resources that limitador would need to evaluate. - * With a shared domain comes the risk of a clash. To avoid a potential clash, either the user or - Kuadrant operator would need to inject a globally unique condition into each rate limit - resource. -* *Limited ability to invoke rate limiting based on the method or path*: Although Envoy supports -applying rate limits at both the virtual host and also the route level, via Istio this currently -only works if you are using a VirtualService. This is because the EnvoyFilter needed to configure -rate limiting needs a -[named route](https://istio.io/latest/docs/reference/config/networking/envoy-filter/#EnvoyFilter-RouteConfigurationMatch-RouteMatch) -in order to match and apply a change to a specific route. This means for non VirtualService routing -(IE HTTPRoute) path, header and method conditional rules must all be applied in Limitador directly -which naturally creates additional load on Limitador, latency for endpoints that don’t need/want -rate limiting and the descriptors needed to apply rate limiting rules must all be defined at the -host level rather than based on the path / method. Issues capturing this limitation are linked -below: - * https://github.com/istio/istio/issues/36790 - * https://github.com/istio/istio/issues/37346 - * https://github.com/kubernetes-sigs/gateway-api/pull/996 - -Therefore, not giving up entirely in existing -[Envoy's RateLimit Filter](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/ratelimit/v3/rate_limit.proto#extension-envoy-filters-network-ratelimit), -we decided to move on and leverage the Envoy's -[Wasm Network Filter](https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/wasm_filter) -and implement rate limiting [wasm-shim](https://github.com/Kuadrant/wasm-shim) -module compliant with the Envoy's -[Rate Limit Service (RLS)](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ratelimit/v3/rls.proto). -This wasm-shim module accepts a [PluginConfig](https://github.com/Kuadrant/kuadrant-operator/blob/fa2b52967409b7c4ea2c2e3412ecf80a8ad2b802/pkg/istio/wasm.go#L24) -struct object as input configuration object. - -WASM filter configuration object ([PluginConfig](https://github.com/Kuadrant/wasm-shim/blob/0b8a12a66fd0d511cb487338f0eb5d9d021fb082/src/configuration.rs) struct): +**Motivation:** _Fine-grained matching rules_
+A second limitation of configuring the rate limit filter via Istio, particularly from [Gateway API](https://gateway-api.sigs.k8s.io) resources, is that [rate limit descriptors](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/rate_limit_filter#composing-actions) at the level of a specific HTTP route rule require "named routes" – defined only in an Istio [VirtualService](https://istio.io/latest/docs/reference/config/networking/virtual-service/#HTTPRoute) resource and referred in an [EnvoyFilter](https://istio.io/latest/docs/reference/config/networking/envoy-filter/#EnvoyFilter-RouteConfigurationMatch-RouteMatch) one. Because Gateway API HTTPRoute rules lack a "name" property[^1], as well as the Istio VirtualService resources are only ephemeral data structures handled by Istio in-memory in its implementation of gateway configuration for Gateway API, where the names of individual route rules are auto-generated and not referable by users in a policy[^2][^3], rate limiting by attributes of the HTTP request (e.g., path, method, headers, etc) would be very limited while depending only on Envoy's [Rate Limit](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/rate_limit_filter) filter. -```yaml -# The filter’s behaviour in case the rate limiting service does not respond back. When it is set to true, Envoy will not allow traffic in case of communication failure between rate limiting service and the proxy. -failure_mode_deny: true -rate_limit_policies: - - name: toystore - rate_limit_domain: toystore-app - upstream_cluster: rate-limit-cluster - hostnames: ["*.toystore.com"] - gateway_actions: - - rules: - - paths: ["/admin/toy"] - methods: ["GET"] - hosts: ["pets.toystore.com"] - configurations: - - actions: - - generic_key: - descriptor_key: admin - descriptor_value: "1" -``` +[^1]: https://github.com/kubernetes-sigs/gateway-api/pull/996 +[^2]: https://github.com/istio/istio/issues/36790 +[^3]: https://github.com/istio/istio/issues/37346 -The WASM filter configuration resources are part of the internal configuration -and therefore not exposed to the end user. +Motivated by the desire to support multiple rate limit domains per ingress gateway, as well as fine-grained HTTP route matching rules for rate limiting, Kuadrant implements a [wasm-shim](https://github.com/Kuadrant/wasm-shim) that handles the rules to invoke the rate limiting service, complying with Envoy's [Rate Limit Service (RLS)](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ratelimit/v3/rls.proto) protocol. -At the WASM filter level, there are no HTTPRoute level or Gateway level rate limit policies. -The rate limit policies in the wasm plugin configuration may not map 1:1 to -user managed RateLimitPolicy custom resources. WASM rate limit policies have an internal logical -name and a set of hostnames to activate it based on the incoming request’s host header. +The wasm module integrates with the gateway in the data plane via [Wasm Network](https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/wasm_filter) filter, and parses a configuration composed out of user-defined RateLimitPolicy resources by the Kuadrant control plane. Whereas the rate limiting service ("Limitador") remains an implementation of Envoy's RLS protocol, capable of being integrated directly via [Rate Limit](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/ratelimit/v3/rate_limit.proto#extension-envoy-filters-network-ratelimit) extension or by Kuadrant, via wasm module for the [Istio Gateway API implementation](https://gateway-api.sigs.k8s.io/implementations/#istio). -Kuadrant deploys one WASM filter for rate limiting per gateway. Only when rate limiting needs to be -applied in a gateway. +As a consequence of this design: +- Users can define fine-grained rate limit rules that match their Gateway and HTTPRoute definitions including for subsections of these. +- Rate limit definitions are insulated, not leaking across unrelated policies or applications. +- Conditions to activate limits are evaluated in the context of the gateway process, reducing the gRPC calls to the external rate limiting service only to the cases where rate limit counters are known in advance to have to be checked/incremented. +- The rate limiting service can rely on the indexation to look up for groups of limit definitions and counters. +- Components remain compliant with industry protocols and flexible for different integration options. -The WASM filter builds a tree based data structure holding the rate limit policies. -The longest (sub)domain match is used to select the policy to be applied. -Only one policy is being applied per invocation. +A Kuadrant wasm-shim configuration for a composition of RateLimitPolicy custom resources looks like the following and it is generated automatically by the Kuadrant control plane: + +```yaml +apiVersion: extensions.istio.io/v1alpha1 +kind: WasmPlugin +metadata: + name: kuadrant-istio-ingressgateway + namespace: istio-system + … +spec: + phase: STATS + pluginConfig: + failureMode: deny + rateLimitPolicies: + - domain: istio-system/gw-rlp # allows isolating policy rules and improve performance of the rate limit service + hostnames: + - '*.website' + - '*.io' + name: istio-system/gw-rlp + rules: # match rules from the gateway and according to conditions specified in the rlp + - conditions: + - allOf: + - operator: startswith + selector: request.url_path + value: / + data: + - static: # tells which rate limit definitions and counters to activate + key: limit.internet_traffic_all__593de456 + value: "1" + - conditions: + - allOf: + - operator: startswith + selector: request.url_path + value: / + - operator: endswith + selector: request.host + value: .io + data: + - static: + key: limit.internet_traffic_apis_per_host__a2b149d2 + value: "1" + - selector: + selector: request.host + service: kuadrant-rate-limiting-service + - domain: default/app-rlp + hostnames: + - '*.toystore.website' + - '*.toystore.io' + name: default/app-rlp + rules: # matches rules from a httproute and additional specified in the rlp + - conditions: + - allOf: + - operator: startswith + selector: request.url_path + value: /assets/ + data: + - static: + key: limit.toystore_assets_all_domains__8cfb7371 + value: "1" + - conditions: + - allOf: + - operator: startswith + selector: request.url_path + value: /v1/ + - operator: eq + selector: request.method + value: GET + - operator: endswith + selector: request.host + value: .toystore.website + - operator: eq + selector: auth.identity.username + value: "" + - allOf: + - operator: startswith + selector: request.url_path + value: /v1/ + - operator: eq + selector: request.method + value: POST + - operator: endswith + selector: request.host + value: .toystore.website + - operator: eq + selector: auth.identity.username + value: "" + data: + - static: + key: limit.toystore_v1_website_unauthenticated__3f9c40c6 + value: "1" + service: kuadrant-rate-limiting-service + selector: + matchLabels: + istio.io/gateway-name: istio-ingressgateway + url: oci://quay.io/kuadrant/wasm-shim:v0.3.0 +``` diff --git a/doc/ratelimitpolicy-reference.md b/doc/ratelimitpolicy-reference.md index cb1fab1c7..e5ee83ebc 100644 --- a/doc/ratelimitpolicy-reference.md +++ b/doc/ratelimitpolicy-reference.md @@ -1,87 +1,71 @@ # The RateLimitPolicy Custom Resource Definition (CRD) - -* [The RateLimitPolicy Custom Resource Definition (CRD)](#the-ratelimitpolicy-custom-resource-definition-crd) - * [RateLimitPolicy](#ratelimitpolicy) - * [RateLimitPolicySpec](#ratelimitpolicyspec) - * [RateLimit](#ratelimit) - * [Configuration](#configuration) - * [ActionSpecifier](#actionspecifier) - * [Rule](#rule) - * [Limit](#limit) - * [RateLimitPolicyStatus](#ratelimitpolicystatus) - * [ConditionSpec](#conditionspec) - - - -Generated using [github-markdown-toc](https://github.com/ekalinin/github-markdown-toc) +- [RateLimitPolicy](#ratelimitpolicy) +- [RateLimitPolicySpec](#ratelimitpolicyspec) + - [Limit](#limit) + - [RateLimit](#ratelimit) + - [RouteSelector](#routeselector) + - [WhenCondition](#whencondition) +- [RateLimitPolicyStatus](#ratelimitpolicystatus) + - [ConditionSpec](#conditionspec) ## RateLimitPolicy -| **json/yaml field** | **Type** | **Required** | **Description** | -|---------------------|-------------------------------------------------|--------------|------------------------------------------------------| -| `spec` | [RateLimitPolicySpec](#RateLimitPolicySpec) | Yes | The specfication for RateLimitPolicy custom resource | -| `status` | [RateLimitPolicyStatus](#RateLimitPolicyStatus) | No | The status for the custom resource | +| **Field** | **Type** | **Required** | **Description** | +|-----------|-------------------------------------------------|:------------:|------------------------------------------------------| +| `spec` | [RateLimitPolicySpec](#ratelimitpolicyspec) | Yes | The specfication for RateLimitPolicy custom resource | +| `status` | [RateLimitPolicyStatus](#ratelimitpolicystatus) | No | The status for the custom resource | ## RateLimitPolicySpec -| **json/yaml field** | **Type** | **Required** | **Default value** | **Description** | -|---------------------|------------------------------------------------------------------------------------------------------------------------------------|--------------|-------------------|---------------------------------------------| -| `targetRef` | [gatewayapiv1alpha2.PolicyTargetReference](https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1alpha2/policy_types.go) | Yes | N/A | identifies an API object to apply policy to | -| `rateLimits` | [][RateLimit](#RateLimit) | No | empy list | list of rate limit configurations | +| **Field** | **Type** | **Required** | **Description** | +|-------------|---------------------------------------------------------------------------------------------------------------------------------------------|--------------|----------------------------------------------------------------| +| `targetRef` | [PolicyTargetReference](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1alpha2.PolicyTargetReference) | Yes | Reference to a Kuberentes resource that the policy attaches to | +| `limits` | Map | No | Limit definitions | -### RateLimit +### Limit -| **json/yaml field** | **Type** | **Required** | **Default value** | **Description** | -|---------------------|-----------------------------------|--------------|---------------------------------|---------------------------------------------------------------------------------------------------------------------| -| `configurations` | [][Configuration](#Configuration) | No | Empty | list of action configurations | -| `rules` | [][Rule](#Rule) | No | Empty. All configurations apply | list of action configurations rules. Rate limit configuration will apply when at least one rule matches the request | -| `limits` | [][Limit](#Limit) | No | Empty | list of Limitador limit objects | +| **Field** | **Type** | **Required** | **Description** | +|------------------|-----------------------------------|:------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `rates` | [][RateLimit](#ratelimit) | No | List of rate limits associated with the limit definition | +| `counters` | []String | No | List of rate limit counter qualifiers. Items must be a valid [Well-known attribute](https://github.com/Kuadrant/architecture/blob/main/rfcs/0002-well-known-attributes.md). Each distinct value resolved in the data plane starts a separate counter for each rate limit. | +| `routeSelectors` | [][RouteSelector](#routeselector) | No | List of selectors of HTTPRouteRules whose matching rules activate the limit. At least one HTTPRouteRule must be selected to activate the limit. If omitted, all HTTPRouteRules of the targeted HTTPRoute activate the limit. Do not use it in policies targeting a Gateway. | +| `when` | [][WhenCondition](#whencondition) | No | List of additional dynamic conditions (expressions) to activate the limit. All expression must evaluate to true for the limit to be applied. Use it for filterring attributes that cannot be expressed in the targeted HTTPRoute's `spec.hostnames` and `spec.rules.matches` fields, or when targeting a Gateway. | -#### Configuration +#### RateLimit -| **json/yaml field** | **Type** | **Required** | **Default value** | **Description** | -|---------------------|---------------------------------------|--------------|-------------------|-----------------------------------------------------------------------------------| -| `actions` | [][ActionSpecifier](#ActionSpecifier) | No | empty | list of action specifiers. Each action specifier can only define one action type. | +| **Field** | **Type** | **Required** | **Description** | +|------------------|----------|:------------:|----------------------------------------------------------------------------------------| +| `limit` | Number | Yes | Maximum value allowed within the given period of time (duration) | +| `duration` | Number | Yes | The period of time in the specified unit that the limit applies | +| `unit` | String | Yes | Unit of time for the duration of the limit. One-of: "second", "minute", "hour", "day". | -#### ActionSpecifier +#### RouteSelector -| **json/yaml field** | **Type** | **Required** | **Default value** | **Description** | -|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|-------------------|----------------------------------------------------------------------------------------------------------------| -| `generic_key` | [config.route.v3.RateLimit.Action.GenericKey](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#config-route-v3-ratelimit-action-generickey) | No | null | generic key action | -| `metadata` | [config.route.v3.RateLimit.Action.MetaData](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#config-route-v3-ratelimit-action-metadata) | No | null | descriptor entry is appended when the metadata contains a key value | -| `remote_address` | [config.route.v3.RateLimit.Action.RemoteAddress](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#config-route-v3-ratelimit-action-remoteaddress) | No | null | descriptor entry is appended to the descriptor and is populated using the trusted address from x-forwarded-for | -| `request_headers` | [config.route.v3.RateLimit.Action.RequestHeaders](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#config-route-v3-ratelimit-action-requestheaders) | No | null | descriptor entry is appended when a header contains a key that matches the header_name | +| **Field** | **Type** | **Required** | **Description** | +|-------------|--------------------------------------------------------------------------------------------------------------------------------|:------------:|-----------------------------------------------------------------------------| +| `hostnames` | [][Hostname](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1beta1.Hostname) | No | List of hostnames of the HTTPRoute that activate the limit | +| `matches` | [][HTTPRouteMatch](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPRouteMatch) | No | List of selectors of HTTPRouteRules whose matching rules activate the limit | -#### Rule +Check out _Kuadrant Rate Limiting > [Route selectors](rate-limiting.md#route-selectors)_ for the semantics of how route selectors work. -| **json/yaml field** | **Type** | **Required** | **Default value** | **Description** | -|---------------------|----------|--------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `paths` | []string | No | null | list of paths. Request matches when one from the list matches | -| `methods` | []string | No | null | list of methods to match. Request matches when one from the list matches | -| `hosts` | []string | No | null | list of hostnames to match. Wildcard hostnames are valid. Request matches when one from the list matches. Each defined hostname must be subset of one of the hostnames defined by the targeted network resource | +#### WhenCondition -#### Limit - -| **json/yaml field** | **Type** | **Required** | **Default value** | **Description** | -|---------------------|----------|--------------|-------------------|-------------------------------------------------------------------------------------------------| -| `maxValue` | int | Yes | N/A | max number of request for the specified time period | -| `seconds` | int | Yes | N/A | time period in seconds | -| `conditions` | []string | No | Empty list | Limit conditions. Check [Limitador](https://github.com/Kuadrant/limitador) for more information | -| `variables` | []string | No | Empty list | Limit variables. Check [Limitador](https://github.com/Kuadrant/limitador) for more information | +| **Field** | **Type** | **Required** | **Description** | +|------------|----------|:------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `selector` | String | Yes | A valid [Well-known attribute](https://github.com/Kuadrant/architecture/blob/main/rfcs/0002-well-known-attributes.md) whose resolved value in the data plane will be compared to `value`, using the `operator`. | +| `operator` | String | Yes | The binary operator to be applied to the resolved value specified by the selector. One-of: "eq" (equal to), "neq" (not equal to) | +| `value` | String | Yes | The static value to be compared to the one resolved from the selector. | ## RateLimitPolicyStatus -| **json field** | **Type** | **Info** | -|----------------------|---------------------------------------|----------------------------------------------------------------------------| -| `observedGeneration` | string | helper field to see if status info is up to date with latest resource spec | -| `conditions` | array of [condition](#ConditionSpec)s | resource conditions | +| **Field** | **Type** | **Description** | +|----------------------|-----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------| +| `observedGeneration` | String | Number of the last observed generation of the resource. Use it to check if the status info is up to date with latest resource spec. | +| `conditions` | [][ConditionSpec](#conditionspec) | List of conditions that define that status of the resource. | ### ConditionSpec -The status object has an array of Conditions through which the resource has or has not passed. -Each element of the Condition array has the following fields: - * The *lastTransitionTime* field provides a timestamp for when the entity last transitioned from one status to another. * The *message* field is a human-readable message indicating details about the transition. * The *reason* field is a unique, one-word, CamelCase reason for the condition’s last transition. @@ -89,10 +73,10 @@ Each element of the Condition array has the following fields: * The *type* field is a string with the following possible values: * Available: the resource has successfully configured; -| **Field** | **json field** | **Type** | **Info** | -|--------------------|----------------------|-----------|------------------------------| -| Type | `type` | string | Condition Type | -| Status | `status` | string | Status: True, False, Unknown | -| Reason | `reason` | string | Condition state reason | -| Message | `message` | string | Condition state description | -| LastTransitionTime | `lastTransitionTime` | timestamp | Last transition timestamp | +| **Field** | **Type** | **Description** | +|----------------------|-----------|------------------------------| +| `type` | String | Condition Type | +| `status` | String | Status: True, False, Unknown | +| `reason` | String | Condition state reason | +| `message` | String | Condition state description | +| `lastTransitionTime` | Timestamp | Last transition timestamp | diff --git a/doc/user-guides/authenticated-rl-for-api-owners.md b/doc/user-guides/authenticated-rl-for-api-owners.md deleted file mode 100644 index 124966a85..000000000 --- a/doc/user-guides/authenticated-rl-for-api-owners.md +++ /dev/null @@ -1,220 +0,0 @@ -# Authenticated Rate Limit For API Owners - -This user guide shows how to configure authenticated rate limiting. -Authenticated rate limiting allows to specify rate limiting configurations -based on the traffic owners, i.e. ID of the user owning the request. -Authentication method used will be the API key. - -### Clone the project - -```sh -git clone https://github.com/Kuadrant/kuadrant-operator && cd kuadrant-operator -``` - -### Setup environment - -This step creates a containerized Kubernetes server locally using [Kind](https://kind.sigs.k8s.io), -then it installs Istio, Kubernetes Gateway API and kuadrant. - -```sh -make local-setup -``` - -### Apply Kuadrant CR - -```sh -kubectl -n kuadrant-system apply -f - < + +Authenticated rate limiting rate limits the traffic directed to an application based on attributes of the client user, who is authenticated by some authentication method. A few examples of authenticated rate limiting use cases are: +- User A can send up to 50rps ("requests per second"), while User B can send up to 100rps. +- Each user can send up to 20rpm ("request per minute"). +- Admin users (members of the 'admin' group) can send up to 100rps, while regular users (non-admins) can send up to 20rpm and no more than 5rps. + +
+ +In this guide, we will rate limit a sample REST API called **Toy Store**. In reality, this API is just an echo service that echoes back to the user whatever attributes it gets in the request. The API exposes an endpoint at `GET http://api.toystore.com/toy`, to mimic an operation of reading toy records. + +We will define 2 users of the API, which can send requests to the API at different rates, based on their user IDs. The authentication method used is **API key**. + +| User ID | Rate limit | +|---------|----------------------------------------| +| alice | 5rp10s ("5 requests every 10 seconds") | +| bob | 2rp10s ("2 requests every 10 seconds") | + +
+ +## Run the steps ① → ④ + +### ① Setup + +This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using [Kind](https://kind.sigs.k8s.io), +where it installs Istio, Kubernetes Gateway API and Kuadrant itself. + +> **Note:** In production environment, these steps are usually performed by a cluster operator with administrator privileges over the Kubernetes cluster. + +Clone the project: + +```sh +git clone https://github.com/Kuadrant/kuadrant-operator && cd kuadrant-operator +``` + +Setup the environment: + +```sh +make local-setup +``` + +Request an instance of Kuadrant: + +```sh +kubectl -n kuadrant-system apply -f - < **Note**: If the command above fails to hit the Toy Store API on your environment, try forwarding requests to the service: +> +> ```sh +> kubectl port-forward -n istio-system service/istio-ingressgateway 9080:80 2>&1 >/dev/null & +> ``` + +### ③ Enforce authentication on requests to the Toy Store API + +Create a Kuadrant `AuthPolicy` to configure the authentication: + +```sh +kubectl apply -f - < **Note:** Kuadrant stores API keys as Kubernetes Secret resources. User metadata can be stored in the annotations of the resource. + +```sh +kubectl apply -f - < **Note:** It may take a couple of minutes for the RateLimitPolicy to be applied depending on your cluster. + +
+ +Verify the rate limiting works by sending requests as Alice and Bob. + +Up to 5 successful (`200 OK`) requests every 10 seconds allowed for Alice, then `429 Too Many Requests`: + +```sh +while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' -H 'Host: api.toystore.com' http://localhost:9080/toy | egrep --color "\b(429)\b|$"; sleep 1; done +``` + +Up to 2 successful (`200 OK`) requests every 10 seconds allowed for Bob, then `429 Too Many Requests`: + +```sh +while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H 'Authorization: APIKEY IAMBOB' -H 'Host: api.toystore.com' http://localhost:9080/toy | egrep --color "\b(429)\b|$"; sleep 1; done +``` + +## Cleanup + +```sh +make local-cleanup +``` diff --git a/doc/user-guides/authenticated-rl-with-jwt-and-k8s-authnz.md b/doc/user-guides/authenticated-rl-with-jwt-and-k8s-authnz.md index 51a7182c3..51182633a 100644 --- a/doc/user-guides/authenticated-rl-with-jwt-and-k8s-authnz.md +++ b/doc/user-guides/authenticated-rl-with-jwt-and-k8s-authnz.md @@ -1,33 +1,69 @@ -# Rate-limiting and protecting an API with JSON Web Tokens (JWTs) and Kubernetes authnz using Kuadrant +# Authenticated Rate Limiting with JWTs and Kubernetes RBAC -Example of rate-limiting and protecting an API (the Toy Store API) with authentication based on ID tokens (signed JWTs) -issued by an OpenId Connect (OIDC) server (Keycloak) and alternative Kubernetes Service Account tokens, and authorization -based on Kubernetes RBAC, with permissions (bindings) stored as Kubernetes Roles and RoleBindings. +This user guide walks you through an example of how to use Kuadrant to protect an application with policies to enforce: +- authentication based OpenId Connect (OIDC) ID tokens (signed JWTs), issued by a Keycloak server; +- alternative authentication method by Kubernetes Service Account tokens; +- authorization delegated to Kubernetes RBAC system; +- rate limiting by user ID. -## Pre-requisites +
+ +In this example, we will protect a sample REST API called **Toy Store**. In reality, this API is just an echo service that echoes back to the user whatever attributes it gets in the request. + +The API listens to requests at the hostnames `*.toystore.com`, where it exposes the endpoints `GET /toy*`, `POST /admin/toy` and `DELETE /amind/toy`, respectively, to mimic operations of reading, creating, and deleting toy records. + +Any authenticated user/service account can send requests to the Toy Store API, by providing either a valid Keycloak-issued access token or Kubernetes token. + +Privileges to execute the requested operation (read, create or delete) will be granted according to the following RBAC rules, stored in the Kubernetes authorization system: + +| Operation | Endpoint | Required role | +|-----------|---------------------|-------------------| +| Read | `GET /toy*` | `toystore-reader` | +| Create | `POST /admin/toy` | `toystore-write` | +| Delete | `DELETE /admin/toy` | `toystore-write` | + +Each user will be entitled to a maximum of 5rp10s (5 requests every 10 seconds). + +## Requirements - [Docker](https://www.docker.com/) - [kubectl](https://kubernetes.io/docs/reference/kubectl/) command-line tool - [jq](https://stedolan.github.io/jq/) -## Run the guide ❶ → ❽ +## Run the guide ① → ⑥ + +### ① Setup a cluster with Kuadrant + +This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using [Kind](https://kind.sigs.k8s.io), +where it installs Istio, Kubernetes Gateway API and Kuadrant itself. -### ❶ Clone the project +> **Note:** In production environment, these steps are usually performed by a cluster operator with administrator privileges over the Kubernetes cluster. + +Clone the project: ```sh git clone https://github.com/Kuadrant/kuadrant-operator && cd kuadrant-operator ``` -### ❷ Setup environment - -This step creates a containerized Kubernetes server locally using [Kind](https://kind.sigs.k8s.io), -then it installs Istio, Kubernetes Gateway API and kuadrant. +Setup the environment: ```sh make local-setup ``` -### ❸ Deploy the API +Request an instance of Kuadrant: + +```sh +kubectl -n kuadrant-system apply -f - < **Note**: If the command above fails to hit the Toy Store API on your environment, try forwarding requests to the service: +> +> ```sh +> kubectl port-forward -n istio-system service/istio-ingressgateway 9080:80 2>&1 >/dev/null & +> ``` -```bash -kubectl port-forward -n istio-system service/istio-ingressgateway 9080:80 & -``` - -### ❹ Request the Kuadrant instance - -```sh -kubectl -n kuadrant-system apply -f - < **Note:** The Keycloak server may take a couple of minutes to be ready. -The Keycloak server may take a couple of minutes to be ready. +### ④ Enforce authentication and authorization for the Toy Store API -### ❻ Create the `AuthPolicy` +Create a Kuadrant `AuthPolicy` to configure authentication and authorization: ```sh kubectl apply -f - < - Can I use Roles and RoleBindings instead of ClusterRoles and ClusterRoleBindings? + Q: Can I use Roles and RoleBindings instead of ClusterRoles and ClusterRoleBindings? Yes, you can. @@ -267,7 +293,7 @@ curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Host: api.toystore.com' -X POS # HTTP/1.1 200 OK ``` -Send requests to the API as the service account (missing permission): +Send requests to the API as the Kubernetes service account: ```sh curl -H "Authorization: Bearer $SA_TOKEN" -H 'Host: api.toystore.com' http://localhost:9080/toy -i @@ -279,38 +305,29 @@ curl -H "Authorization: Bearer $SA_TOKEN" -H 'Host: api.toystore.com' -X POST ht # HTTP/1.1 403 Forbidden ``` -### ❽ Create the `RateLimitPolicy` +### ⑥ Enforce rate limiting on requests to the Toy Store API + +Create a Kuadrant `RateLimitPolicy` to configure rate limiting: ```sh kubectl apply -f - < **Note:** If the tokens have expired, you may need to refresh them first. + Send requests as the Keycloak-authenticated user: ```sh while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Host: api.toystore.com' http://localhost:9080/toy | egrep --color "\b(429)\b|$"; sleep 1; done ``` -Send requests as the service account: +Send requests as the Kubernetes service account: ```sh while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Authorization: Bearer $SA_TOKEN" -H 'Host: api.toystore.com' http://localhost:9080/toy | egrep --color "\b(429)\b|$"; sleep 1; done ``` -Each user should be entitled to a maximum of 5 requests to the API every 10 seconds. - -> **Note:** You may need to refresh the tokens if they are expired. - ## Cleanup ```sh diff --git a/doc/user-guides/gateway-rl-for-cluster-operators.md b/doc/user-guides/gateway-rl-for-cluster-operators.md index af5bb56c1..26651ce50 100644 --- a/doc/user-guides/gateway-rl-for-cluster-operators.md +++ b/doc/user-guides/gateway-rl-for-cluster-operators.md @@ -1,244 +1,204 @@ -# Gateway Rate Limit For Cluster Operators +# Gateway Rate Limiting for Cluster Operators -This user guide shows how the kuadrant's control plane applies rate limit policy at -[Gateway API's Gateway](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1beta1.Gateway) -level. +This user guide walks you through an example of how to configure rate limiting for all routes attached to an ingress gateway. -### Clone the project +
+ +## Run the steps ① → ⑤ + +### ① Setup + +This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using [Kind](https://kind.sigs.k8s.io), +where it installs Istio, Kubernetes Gateway API and Kuadrant itself. + +> **Note:** In production environment, these steps are usually performed by a cluster operator with administrator privileges over the Kubernetes cluster. + +Clone the project: ```sh git clone https://github.com/Kuadrant/kuadrant-operator && cd kuadrant-operator ``` -### Setup environment - -This step creates a containerized Kubernetes server locally using [Kind](https://kind.sigs.k8s.io), -then it installs Istio, Kubernetes Gateway API and kuadrant. +Setup the environment: ```sh make local-setup ``` -### Apply Kuadrant CR +Request an instance of Kuadrant: ```sh kubectl -n kuadrant-system apply -f - < **Note:** It may take a couple of minutes for the RateLimitPolicy to be applied depending on your cluster. -`GET /toy` @ **1** rps (expected to be rate limited @ **8** reqs / **10** secs (0.8 rps)) -```sh -while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Host: api.toystore.com" http://localhost:9080/toy | egrep --color "\b(429)\b|$"; sleep 1; done -``` +### ④ Deploy a sample API to test rate limiting enforced at the level of the gateway -`POST /admin/toy` @ **1** rps (expected to be rate limited @ **5** reqs / **10** secs (0.5 rps)) -```sh -while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Host: api.toystore.com" -X POST http://localhost:9080/admin/toy | egrep --color "\b(429)\b|$"; sleep 1; done +``` + ┌───────────┐ ┌───────────┐ +┌───────────────────┐ │ (Gateway) │ │ (Gateway) │ +│ (RateLimitPolicy) │ │ external │ │ internal │ +│ gw-rlp ├─────►│ │ │ │ +└───────────────────┘ │ *.io │ │ *.local │ + └─────┬─────┘ └─────┬─────┘ + │ │ + └─────────┬────────┘ + │ + ┌─────────┴────────┐ + │ (HTTPRoute) │ + │ toystore │ + │ │ + │ *.toystore.io │ + │ *.toystore.local │ + └────────┬─────────┘ + │ + ┌──────┴───────┐ + │ (Service) │ + │ toystore │ + └──────────────┘ ``` -### Rate limiting Gateway traffic - -![](https://i.imgur.com/0o3yQzP.png) +Deploy the sample API: -RateLimitPolicy applied for the Gateway. +```sh +kubectl apply -f examples/toystore/toystore.yaml +``` -| Policy | Rate Limits | -|---------------|------------------------------------:| -| `POST /*` | **2** reqs / **10** secs (0.2 rps) | -| Per remote IP | **25** reqs / **10** secs (2.5 rps) | +Route traffic to the API from both gateways: ```sh kubectl apply -f - <&1 >/dev/null & +kubectl port-forward -n istio-system service/internal-istio 9082:80 2>&1 >/dev/null & ``` -`POST /admin/toy` @ **1** rps (expected to be rate limited @ **2** reqs / **10** secs (0.2 rps)) +Up to 5 successful (`200 OK`) requests every 10 seconds through the `external` ingress gateway (`*.io`), then `429 Too Many Requests`: + ```sh -while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Host: api.toystore.com" -X POST http://localhost:9080/admin/toy | egrep --color "\b(429)\b|$"; sleep 1; done +while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H 'Host: api.toystore.io' http://localhost:9081 | egrep --color "\b(429)\b|$"; sleep 1; done ``` -### Validating Gateway "Per Remote IP" policy +Unlimited successful (`200 OK`) through the `internal` ingress gateway (`*.local`): -Stop all traffic. +```sh +while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H 'Host: api.toystore.local' http://localhost:9082 | egrep --color "\b(429)\b|$"; sleep 1; done +``` -`GET /free` @ **3** rps (expected to be rate limited @ **25** reqs / **10** secs (2.5 rps)) +## Cleanup ```sh -while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null -H "Host: api.toystore.com" http://localhost:9080/free -: --write-out '%{http_code}\n' --silent --output /dev/null -H "Host: api.toystore.com" http://localhost:9080/free -: --write-out '%{http_code}\n' --silent --output /dev/null -H "Host: api.toystore.com" http://localhost:9080/free | egrep --color "\b(429)\b|$"; sleep 1; done +make local-cleanup ``` diff --git a/doc/user-guides/simple-rl-for-api-owners.md b/doc/user-guides/simple-rl-for-api-owners.md deleted file mode 100644 index dc73b3c0c..000000000 --- a/doc/user-guides/simple-rl-for-api-owners.md +++ /dev/null @@ -1,137 +0,0 @@ -# Simple Rate Limit For API Owners - -This user guide shows how to configure rate limiting for one of the subdomains. - -### Clone the project - -```sh -git clone https://github.com/Kuadrant/kuadrant-operator -``` - -### Setup environment - -This step creates a containerized Kubernetes server locally using [Kind](https://kind.sigs.k8s.io), -then it installs Istio, Kubernetes Gateway API and kuadrant. - -```sh -make local-setup -``` - -### Apply Kuadrant CR - -```sh -kubectl -n kuadrant-system apply -f - < + +In this guide, we will rate limit a sample REST API called **Toy Store**. In reality, this API is just an echo service that echoes back to the user whatever attributes it gets in the request. The API listens to requests at the hostname `api.toystore.com`, where it exposes the endpoints `GET /toys*` and `POST /toys`, respectively, to mimic a operations of reading and writing toy records. + +We will rate limit the `POST /toys` endpoint to a maximum of 5rp10s ("5 requests every 10 seconds"). + +
+ +## Run the steps ① → ③ + +### ① Setup + +This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using [Kind](https://kind.sigs.k8s.io), +where it installs Istio, Kubernetes Gateway API and Kuadrant itself. + +> **Note:** In production environment, these steps are usually performed by a cluster operator with administrator privileges over the Kubernetes cluster. + +Clone the project: + +```sh +git clone https://github.com/Kuadrant/kuadrant-operator && cd kuadrant-operator +``` + +Setup the environment: + +```sh +make local-setup +``` + +Request an instance of Kuadrant: + +```sh +kubectl -n kuadrant-system apply -f - < **Note**: If the command above fails to hit the Toy Store API on your environment, try forwarding requests to the service: +> +> ```sh +> kubectl port-forward -n istio-system service/istio-ingressgateway 9080:80 2>&1 >/dev/null & +> ``` + +### ③ Enforce rate limiting on requests to the Toy Store API + +Create a Kuadrant `RateLimitPolicy` to configure rate limiting: + +![](https://i.imgur.com/2A9sXXs.png) + +```sh +kubectl apply -f - < **Note:** It may take a couple of minutes for the RateLimitPolicy to be applied depending on your cluster. + +
+ +Verify the rate limiting works by sending requests in a loop. + +Up to 5 successful (`200 OK`) requests every 10 seconds to `POST /toys`, then `429 Too Many Requests`: + +```sh +while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H 'Host: api.toystore.com' http://localhost:9080/toys -X POST | egrep --color "\b(429)\b|$"; sleep 1; done +``` + +Unlimited successful (`200 OK`) to `GET /toys`: + +```sh +while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H 'Host: api.toystore.com' http://localhost:9080/toys | egrep --color "\b(429)\b|$"; sleep 1; done +``` + +## Cleanup + +```sh +make local-cleanup +``` diff --git a/examples/toystore/ratelimitpolicy_gateway.yaml b/examples/toystore/ratelimitpolicy_gateway.yaml index d1b795bb2..57b53002b 100644 --- a/examples/toystore/ratelimitpolicy_gateway.yaml +++ b/examples/toystore/ratelimitpolicy_gateway.yaml @@ -1,5 +1,4 @@ ---- -apiVersion: kuadrant.io/v1beta1 +apiVersion: kuadrant.io/v1beta2 kind: RateLimitPolicy metadata: name: toystore-gw @@ -9,25 +8,23 @@ spec: group: gateway.networking.k8s.io kind: Gateway name: istio-ingressgateway - rateLimits: - - rules: - - methods: ["POST"] - - methods: ["PUT"] - configurations: - - actions: - - generic_key: - descriptor_key: expensive_op - descriptor_value: "1" - limits: - - conditions: ["expensive_op == '1'"] - maxValue: 2 - seconds: 30 - variables: [] - - configurations: - - actions: - - remote_address: {} - limits: - - conditions: [] - maxValue: 5 - seconds: 30 - variables: ["remote_address"] + limits: + "expensive-operation": + rates: + - limit: 2 + duration: 30 + unit: second + when: + - selector: request.method + operator: eq + value: POST + + "limit-per-ip": + rates: + - limit: 5 + duration: 30 + unit: second + when: + - selector: source.ip + operator: eq + value: source.address diff --git a/examples/toystore/ratelimitpolicy_httproute.yaml b/examples/toystore/ratelimitpolicy_httproute.yaml index 6726fff75..de10f7143 100644 --- a/examples/toystore/ratelimitpolicy_httproute.yaml +++ b/examples/toystore/ratelimitpolicy_httproute.yaml @@ -1,5 +1,4 @@ ---- -apiVersion: kuadrant.io/v1beta1 +apiVersion: kuadrant.io/v1beta2 kind: RateLimitPolicy metadata: name: toystore-httproute @@ -8,58 +7,44 @@ spec: group: gateway.networking.k8s.io kind: HTTPRoute name: toystore - rateLimits: - - rules: - - paths: ["/toy"] - methods: ["GET"] - configurations: - - actions: - - generic_key: - descriptor_key: get_toy - descriptor_value: "yes" - limits: - - conditions: ["get_toy == 'yes'"] - maxValue: 2 - seconds: 30 - variables: [] - - rules: - - paths: ["/admin/toy"] - methods: ["POST", "DELETE"] - configurations: - - actions: - - generic_key: - descriptor_key: admin - descriptor_value: "yes" - - metadata: - descriptor_key: "user_id" - default_value: "no-user" - metadata_key: - key: "envoy.filters.http.ext_authz" - path: - - segment: - key: "ext_auth_data" - - segment: - key: "user_id" - limits: - - conditions: - - "admin == 'yes'" - - "user_id == 'bob'" - maxValue: 2 - seconds: 30 - variables: [] - - conditions: - - "admin == 'yes'" - - "user_id == 'alice'" - maxValue: 4 - seconds: 30 - variables: [] - - configurations: - - actions: - - generic_key: - descriptor_key: vhaction - descriptor_value: "yes" - limits: - - conditions: ["vhaction == 'yes'"] - maxValue: 6 - seconds: 30 - variables: [] + limits: + "global": + rates: + - limit: 6 + duration: 30 + unit: second + + "get-toy": + rates: + - limit: 5 + duration: 30 + unit: second + routeSelectors: + - matches: + - path: + type: Exact + value: "/toy" + method: GET + + "admin-post-or-delete-toy-per-user": + rates: + - limit: 2 + duration: 30 + unit: second + counters: + - metadata.filter_metadata.envoy\.filters\.http\.ext_authz.username + routeSelectors: + - matches: + - path: + type: Exact + value: "/admin/toy" + method: POST + - matches: + - path: + type: Exact + value: "/admin/toy" + method: DELETE + when: + - selector: metadata.filter_metadata.envoy\.filters\.http\.ext_authz.admin + operator: eq + value: "true"