Skip to content

Commit

Permalink
API v5.1 (#67)
Browse files Browse the repository at this point in the history
* 20240701 - dev NGINX App Protect 5 compiler

* 20240701 - dev NGINX App Protect 5 compiler

* 20240730 - v5.1 dev

* 2024073002 - v5.1 dev

* 2024073003 dev 5.1

* 20240731 - 5.1 dev

* 2024080201 - 5.1 dev

* 20240806 - 5.1 dev
Moved resolvers config to include files
Added per-upstream resolver support
FEATURES updated

* 2024080602 - 5.1 dev
Moved HTTP upstreams to separate include files
Bugfixes
Updated postman collection

* 20240807 - 5.1 dev
Moved HTTP and Stream upstream to separated include config file
Added DNS resolver support for stream servers and stream upstreams
Moved .declaration.http.resolvers[] to .declaration.resolvers[]
Postman collection updated for v5.1

* 2024080801 - 5.1 dev
Postman collection updated
NGINX One REST API endpoints updated

* 2024080809 - 5.1 dev
FEATURES updated
Postman collection README updated
API v5.1 USAGE updated

* 2024080902 - 5.1 dev
FEATURES updated

* 2024080903 - 5.1 dev
fabriziofiorucci authored Aug 9, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
1 parent 181c7b4 commit 59ea39a
Showing 30 changed files with 1,029 additions and 607 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -24,4 +24,5 @@ Thumbs.db
/src/v5_0/__pycache__/
/src/v5_1/__pycache__/
/contrib/devportal/redocly/src/__pycache__/
/contrib/app-protect/src/__pycache__/
/venv/
192 changes: 134 additions & 58 deletions FEATURES.md

Large diffs are not rendered by default.

35 changes: 11 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
@@ -2,19 +2,18 @@

[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)

This project provides a set of declarative REST API for [NGINX Instance Manager](https://docs.nginx.com/nginx-management-suite/nim/) and [NGINX One Cloud Console](https://docs.nginx.com/nginx-one/) (currently in early stage).
This project provides a set of declarative REST API for [NGINX Instance Manager](https://docs.nginx.com/nginx-management-suite/nim/) and [NGINX One Cloud Console](https://docs.nginx.com/nginx-one/) (currently in early stage / unstable).

It can be used to manage NGINX Plus configuration lifecycle and to create NGINX Plus configurations using JSON service definitions.

GitOps integration is supported: source of truth is checked for updates (NGINX App Protect policies, TLS certificates, keys and chains/bundles, Swagger/OpenAPI definitions, snippets) and NGINX configurations are automatically kept in sync.

Use cases include:

- Rapid configuration generation and templating
- CI/CD integration with NGINX Instance Manager (instance groups and staged configs) and NGINX One Cloud Console (clusters)
- Integration with NGINX Instance Manager (instance group) and NGINX One Cloud Console (config sync group)
- NGINX App Protect DevSecOps integration (NGINX Instance Manager only)
- API Gateway deployments with automated Swagger / OpenAPI schema import
- API Developer portals zero-touch deployment
- API Developer portals zero-touch deployment (redocly and backstage supported)
- GitOps integration with source of truth support for
- NGINX App Protect WAF policies
- TLS certificates, keys and chains/bundles
@@ -29,9 +28,9 @@ A **blog article** to automate NGINX API Gateway management from OpenAPI schemas
## Supported releases

- NGINX Instance Manager 2.14+
- NGINX One Cloud Console
- NGINX One Cloud Console (early stage / unstable)
- NGINX Plus R30+
- NGINX App Protect WAF 4.8+
- NGINX App Protect WAF 4

## Architecture

@@ -55,10 +54,6 @@ stateDiagram-v2
DEVP: Developer Portal Service
OUTPUT: Output
REDIS: Redis
POST: Generic POST endpoint
CONFIGMAP: Kubernetes ConfigMap
PLAINTEXT: Plaintext
BASE64: Base64-encoded
DevOps --> Pipeline
Pipeline --> INPUT
@@ -69,10 +64,6 @@ stateDiagram-v2
SOT --> NDAPI
NDAPI --> REDIS
REDIS --> NDAPI
OUTPUT --> BASE64
OUTPUT --> POST
OUTPUT --> CONFIGMAP
OUTPUT --> PLAINTEXT
OUTPUT --> NIM
OUTPUT --> N1
NDAPI --> DEVP
@@ -124,7 +115,7 @@ critical If Developer Portal enabled
end
end
NGINX Declarative API Core ->>+ NGINX Instance Manager / NGINX One: Publish staged config to instance group / cluster
NGINX Declarative API Core ->>+ NGINX Instance Manager / NGINX One: Publish staged config to instance group / config sync group
NGINX Instance Manager / NGINX One ->> NGINX: Publish config to NGINX instances
NGINX Instance Manager / NGINX One ->>- NGINX Declarative API Core: Publish outcome
@@ -139,12 +130,8 @@ end

## Output formats

- [X] Plaintext
- [X] JSON-wrapped Base64-encoded
- [X] Kubernetes Configmap
- [X] POST to Generic REST API endpoint
- [X] Output to NGINX Instance Manager 2.14+ imperative REST API (NGINX instance groups)
- [X] Output to NGINX One Cloud Console REST API (NGINX clusters)
- [X] Output to NGINX Instance Manager 2.14+ imperative REST API (instance group)
- [X] Output to NGINX One Cloud Console REST API (config sync group)

## Supported features

@@ -154,9 +141,9 @@ See the [features list](/FEATURES.md)

Usage details and JSON schema are available here:

- [API v5.1](/USAGE-v5.1.md) - development
- [API v5.0](/USAGE-v5.0.md) - latest
- [API v4.2](/USAGE-v4.2.md)
- [API v5.1](/USAGE-v5.1.md) - latest
- [API v5.0](/USAGE-v5.0.md)
- [API v4.2](/USAGE-v4.2.md) - deprecated

A sample Postman collection and usage instructions can be found [here](/contrib/postman)

176 changes: 11 additions & 165 deletions USAGE-v5.1.md
Original file line number Diff line number Diff line change
@@ -3,21 +3,13 @@
Version 5.1 supports:

- [NGINX Instance Manager](https://docs.nginx.com/nginx-management-suite/nim/) 2.14+
- [NGINX One Cloud Console](https://docs.nginx.com/nginx-one/)
- [NGINX One Cloud Console](https://docs.nginx.com/nginx-one/) - early stage / unstable
- [NGINX Plus](https://docs.nginx.com/nginx/) R30+
- [NGINX App Protect WAF](https://docs.nginx.com/nginx-app-protect-waf/) 4.2.0+ with compiled [policy bundles](https://docs.nginx.com/nginx-app-protect-waf/v5/admin-guide/compiler/)
- [NGINX App Protect WAF](https://docs.nginx.com/nginx-app-protect-waf/) 4 with precompiled [policy bundles](https://docs.nginx.com/nginx-app-protect-waf/v5/admin-guide/compiler/)

The JSON schema is self explanatory. See also the [sample Postman collection](/contrib/postman)
The JSON schema is self explanatory. See also the [sample Postman collection](/contrib/postman) for usage examples

- `.output.type` defines how NGINX configuration will be returned:
- *plaintext* - plaintext format
- *json* - JSON-wrapped, base64-encoded
- *configmap* - Kubernetes Configmap in YAML format.
- `.output.configmap.name` must be set to the ConfigMap name
- `.output.configmap.filename` must be set to the NGINX configuration filename
- `.output.configmap.namespace` the optional namespace for the ConfigMap
- *http* - NGINX configuration is POSTed to custom url
- `.output.http.url` the URL to POST the configuration to
- *nms* - NGINX configuration is published as a Staged Config to NGINX Instance Manager
- `.output.nms.url` the NGINX Instance Manager URL
- `.output.nms.username` the NGINX Instance Manager authentication username
@@ -38,165 +30,23 @@ The JSON schema is self explanatory. See also the [sample Postman collection](/c
- `.output.nms.policies[].versions[].displayName` the policy version's display name
- `.output.nms.policies[].versions[].description` the policy version's description
- `.output.nms.policies[].versions[].contents` this can be either base64-encoded or be a HTTP(S) URL that will be fetched dynamically from a source of truth
- *nginxone* - NGINX configuration is published to a NGINX One Cloud Console cluster
- *nginxone* - NGINX configuration is published to a NGINX One Cloud Console config sync group
- `.output.nginxone.url` the NGINX One Cloud Console URL
- `.output.nginxone.namespace` the NGINX One Cloud Console namespace
- `.output.nginxone.token` the authentication token
- `.output.nginxone.cluster` the cluster name
- `.output.nginxone.configsyncgroup` the NGINX One Cloud Console config sync group name
- `.output.nginxone.synctime` **optional**, used for GitOps autosync. When specified and the declaration includes HTTP(S) references to NGINX App Protect policies, TLS certificates/keys/chains, the HTTP(S) endpoints will be checked every `synctime` seconds and if external contents have changed, the updated configuration will automatically be published to NGINX One Cloud Console
- `.output.nginxone.modules` an optional array of NGINX module names (ie. 'ngx_http_app_protect_module', 'ngx_http_js_module','ngx_stream_js_module')
- `.output.nginxone.certificates` an optional array of TLS certificates/keys/chains to be published
- `.output.nginxone.certificates[].type` the item type ('certificate', 'key', 'chain')
- `.output.nginxone.certificates[].name` the certificate/key/chain name with no path/extension (ie. 'test-application')
- `.output.nginxone.certificates[].contents` the content: this can be either base64-encoded or be a HTTP(S) URL that will be fetched dynamically from a source of truth
- `.declaration` describes the NGINX configuration to be created.
- `.declaration` describes the NGINX configuration to be created
- `.declaration.http[]` NGINX HTTP definitions
- `.declaration.layer4[]` NGINX TCP/UDP definitions
- `.declaration.resolvers[]` DNS resolvers definitions

### Locations ###

Locations `.declaration.http.servers[].locations[].uri` match modifiers in `.declaration.http.servers[].locations[].urimatch` can be:

- *prefix* - prefix URI matching
- *exact* - exact URI matching
- *regex* - case sensitive regex matching
- *iregex* - case insensitive regex matching
- *best* - case sensitive regex matching that halts any other location matching once a match is made

### Javascript profiles ###

NGINX Javascript profiles are defined in `.declaration.http.njs[]`:

- `name` - the NJS profile name
- `file.content` - the base64-encoded njs source code or the `http(s)://` URL of the file
- `file.authentication.server[0].profile` - authentication profile name if `file.content` is a URL and the request must be authenticated

### Javascript hooks ###

NGINX Javascript hooks can be used in:

- `.declaration.http.njs`
- Supported hooks:
- `js_preload_object'
- 'js_set`
- `.declaration.http.server[].njs`
- Supported hooks:
- `js_preload_object'
- 'js_set`
- `.declaration.http.server[].location[].njs`
- Supported hooks:
- `js_body_filter'
- 'js_content'
- 'js_header_filter'
- 'js_periodic'
- 'js_preload_object'
- 'js_set`

Hooks invocation is:

```
"njs": [
{
"hook": {
"name": "<HOOK_NAME>",
"parameters": [
{
"name": "<HOOK_PARAMETER_NAME>",
"value": "<HOOK_PARAMETER_VALUE>"
}
]
},
"profile": "<NGINX_JAVASCRIPT_PROFILE>",
"function": "<JAVASCRIPT_FUNCTION_NAME>"
}
]
```

For detailed examples see the [Postman collection](/contrib/postman)

### API Gateway ###

Swagger files and OpenAPI schemas can be used to automatically configure NGINX as an API Gateway. Developer portal creation is supported through [Redocly](https://redocly.com/)

Declaration path `.declaration.http.servers[].locations[].apigateway` defines the API Gateway configuration:

- `openapi_schema` - the base64-encoded schema, or the schema URL. YAML and JSON are supported
- `api_gateway.enabled` - enable/disable API Gateway provisioning
- `api_gateway.strip_uri` - removes the `.declaration.http.servers[].locations[].uri` part of the URI before forwarding requests to the upstream
- `api_gateway.server_url` - the base URL of the upstream server
- `developer_portal.enabled` - enable/disable Developer portal provisioning
- `developer_portal.type` - developer portal type. Currently supported are: `redocly`, `backstage`
- `developer_portal.redocly.*` - Redocly-based developer portal parameters. See the [Postman collection](/contrib/postman)
- `developer_portal.backstage.*` - Backstage-based developer portal parameters. See the [Postman collection](/contrib/postman)
- `authentication` - optional, used to enforce authentication at the API Gateway level
- `authentication.client[]` - authentication profile names
- `authentication.enforceOnPaths` - if set to `true` authentication is enforced on all API endpoints listed under `authentication.paths`. if set to `false` authentication is enforced on all API endpoints but those listed under `authentication.paths`
- `authentication.paths` - paths to enforce authentication
- `authorization[]` - optional, used to enforce authorization
- `authorization[].profile` - authorization profile name
- `authorization[].enforceOnPaths` - if set to `true` authorization is enforced on all API endpoints listed under `authorization.paths`. if set to `false` authorization is enforced on all API endpoints but those listed under `authorization[].paths`
- `authorization[].paths` - paths to enforce authorization
- `rate_limit` - optional, used to enforce rate limiting at the API Gateway level
- `rate_limit.enforceOnPaths` - if set to `true` rate limiting is enforced on all API endpoints listed under `rate_limit.paths`. if set to `false` rate limiting is enforced on all API endpoints but those listed under `rate_limit.paths`

A sample API Gateway declaration to publish the `https://petstore.swagger.io` REST API using:

- REST API endpoint URIs
- HTTP Methods
- Rate limiting on `/user/login`, `/usr/logout` and `/pet/{petId}/uploadImage`
- JWT authentication on `/user/login`, `/usr/logout` and `/pet/{petId}/uploadImage`
- JWT claim-based authorization on `/user/login`, `/usr/logout` and `/pet/{petId}/uploadImage`
- Redocly-based developer portal
- NGINX App Protect WAF security

can be found in the [Postman collection](/contrib/)

### Maps ###

Map entries `.declaration.maps[].entries.keymatch` can be:

- *exact* - exact variable matching
- *regex* - case sensitive regex matching
- *iregex* - case insensitive regex matching

### Snippets ###

Snippets for `http`, `upstream`, `server` and `location` can be specified as:
- base64-encoded content
- HTTP(S) URL of a source of truth to fetch snippet content from. Content on the source of truth must be plaintext
- source of truth authentication is supported through authentication profiles

### Authentication profiles ###

Client and Server authentication profiles can be defined in the declarative json at `.declaration.http.authentication`

```commandline
"authentication": {
"client": [
{
"name": "<PROFILE_NAME>",
"type": "<PROFILE_TYPE>",
"<PROFILE_TYPE>": {
"<PROFILE_KEY>": "<VALUE>",
[...]
}
},
[...]
],
"server": [
{
"name": "<PROFILE_NAME>",
"type": "<PROFILE_TYPE>",
"<PROFILE_TYPE>": {
"<PROFILE_KEY>": "<VALUE>",
[...]
}
},
[...]
]
```

For a list of all supported authentication profile types see the [feature matrix](/FEATURES.md)

### API endpoints ###
### API endpoints

- `POST /v5.1/config/` - Publish a new declaration
- `PATCH /v5.1/config/{config_uid}` - Update an existing declaration
@@ -206,8 +56,4 @@ For a list of all supported authentication profile types see the [feature matrix
- Per-Stream upstream CRUD
- Per-NGINX App Protect WAF policy CRUD
- `GET /v5.1/config/{config_uid}` - Retrieve an existing declaration
- `DELETE /v5.1/config/{config_uid}` - Delete an existing declaration

### Usage Examples ###

A sample Postman collection is available [here](/contrib/postman)
- `DELETE /v5.1/config/{config_uid}` - Delete an existing declaration
53 changes: 53 additions & 0 deletions contrib/app-protect/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# syntax=docker/dockerfile:1
ARG VERSION_TAG
ARG BASE_IMAGE=private-registry.nginx.com/nap/waf-compiler:${VERSION_TAG}
FROM ${BASE_IMAGE}

# Installing packages as root
USER root

ENV DEBIAN_FRONTEND="noninteractive"

# REST API wrapper
WORKDIR /compiler
COPY src src/
# REST API wrapper

RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \
--mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \
apt-get update \
&& apt-get install -y \
apt-transport-https \
lsb-release \
ca-certificates \
wget \
gnupg2 \
ubuntu-keyring \
&& wget -qO - https://cs.nginx.com/static/keys/app-protect-security-updates.key | gpg --dearmor | \
tee /usr/share/keyrings/app-protect-security-updates.gpg >/dev/null \
&& printf "deb [signed-by=/usr/share/keyrings/app-protect-security-updates.gpg] \
https://pkgs.nginx.com/app-protect-security-updates/ubuntu `lsb_release -cs` nginx-plus\n" | \
tee /etc/apt/sources.list.d/nginx-app-protect.list \
&& wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx \
&& apt-get update \
&& apt-get install -y \
app-protect-attack-signatures \
app-protect-bot-signatures \
app-protect-threat-campaigns \
# REST API wrapper
&& apt-get -y install python3 python3-venv \
&& python3 -m venv /compiler/env/ \
&& . /compiler/env/bin/activate \
&& pip3 install --no-cache --upgrade pip setuptools virtualenv \
&& python3 -m pip install --upgrade pip \
&& pip3 install -r /compiler/src/requirements.txt \
# REST API wrapper
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

# non-root default user (UID 101)
USER nginx

# REST API wrapper
ENTRYPOINT [ "/compiler/src/start.sh" ]
# REST API wrapper
21 changes: 21 additions & 0 deletions contrib/app-protect/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# REST API for NGINX App Protect WAF Compiler

This contrib provides a set of REST API to use the NGINX App Protect 5 policy compiler

## REST API Endpoints

- `/v1/compile/policy` - compiles a JSON policy into a bundle
- Method: `POST`
- Payload: `{"global-settings": "<BASE64_ENCODED_GLOBAL_SETTINGS_JSON>", "policy": "<BASE64_ENCODED_POLICY_JSON>", "cookie-protection-seed": "<SEED_VALUE>"}`
- `/v1/compile/logprofile` - compiles a log profile JSON into a bundle
- Method: `POST`
- Payload: `{"logprofile": "<BASE64_ENCODED_LOG_PROFILE_JSON>"}`
- `/v1/bundle/info` - returns details on a compiled bundle
- Method: `POST`
- Payload: `{"bundle": "<BASE64_ENCODED_TGZ_BUNDLE>"}`

Headers required for all endpoints:

```
Content-Type: application/json
```
5 changes: 5 additions & 0 deletions contrib/app-protect/src/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
typing
uvicorn
fastapi
uuid
pyyaml
Loading

0 comments on commit 59ea39a

Please sign in to comment.