diff --git a/Makefile b/Makefile index 74b296d4..fdabdc74 100644 --- a/Makefile +++ b/Makefile @@ -146,7 +146,10 @@ inventory-down-kafka: .PHONY: run # run api locally run: build - $(GO) run main.go serve --config .inventory-api.yaml + $(GO) run main.go serve + +run-help: build + $(GO) run main.go serve --help .PHONY: migrate # run database migrations diff --git a/README.md b/README.md index 4f78d441..d8e87593 100644 --- a/README.md +++ b/README.md @@ -1,173 +1,308 @@ # Common Inventory + This repository implements a common inventory system with eventing. +[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) + +## Table of Contents +- [Development Setup](#development-setup) +- [Example Usage](#example-usage) +- [Configuration](#configuration) +- [Testing](#testing) +- [Contributing](#contributing) + +## Development Setup + +### Prerequisites +- Go 1.23.1+ +- Make + +### Running locally + +When running locally, (.inventory-api.yaml)[./.inventory-api.yaml] file is used. By default, this configuration does the following: +- Exposes the inventory API in `localhost` and using port `8000` for http and port `9000` for grpc. +- Sets authentication mechanism to `allow-unauthenticated`, allowing users to be authenticated with their user-agent value. +- Sets authorization mechanism to `allow-all`. +- Configures eventing mechanism to go to stdout. +- Sets database implementation to sqlite3 and the database file to `inventory.db` +- Configures log level to `INFO`. + +NOTE: You can update the [default settings](./.inventory-api.yaml) file as required to test different scenarios. Refer to the command line help (`make run-help`) +for information on the different parameters. + +1. Clone the repository and navigate to the directory. +2. Install the required dependencies + ```shell + make init + ``` + +3. Build the project + ```shell + make build + ``` + +4. Run the database migration + ```shell + make migrate + ``` + +5. Start the development server + ```shell + make run + ``` + +### Overriding commands in the Makefile + +Due to various alternatives to running some images, we accept some arguments to override certains tools + +#### GO binary -## Setup -```bash -make init +Since there are official instructions on how to [manage multiple installs](https://go.dev/doc/manage-install) +We accept the `GO` parameter when running make. e.g. + +```shell +GO=go1.23.1 make run ``` -## API Changes (check against buf repository) -`make api` -## API Breaking Changes -`make api_breaking` +or -## Build -`make build` +```shell +export GO=go1.23.1 +make run +``` + +#### Podman / Docker + +We will use `podman` if it is installed, and we will fall back to `docker`. You can also specify if you want to ensure a particular binary is used +by providing `DOCKER` parameter e.g. -## Build Container Images -By default the quay repository is `quay.io/cloudservices/kessel-inventory`. If you wish to use another for testing, set IMAGE value first ```shell -export IMAGE=your-quay-repo # if desired -make docker-build-push +DOCKER=docker make api ``` -## Run inventory api locally -### Run migration -`make migrate` -### Run service -`make run` +or +```shell +export DOCKER=docker +make api +``` -## Run docker-compose to setup -```make inventory-up``` to setup inventory-api, relations-api, spicedb, postgres +### Debugging -## Tear down docker-compose -`make inventory-down` +See [DEBUG](./DEBUG.md) for instructions on how to debug + +### Alternatives way of running this service +#### Kessel Inventory + Kessel Relations -## Example Usage +There is Make target to run inventory-api (with postgres) and relations api (with spicedb). +It uses compose to build the current inventory code and spin up containers with the required dependecies. -### Health check endpoints +- This provides a [PSK file](./config/psks.yaml#L1) with a token "1234". +- Default ports in this setup are `8081` for http and `9091` for grpc. +- Refer to [inventory-api-compose.yaml](./inventory-api-compose.yaml) for additional configuration + +To start use: +```shell +make inventory-up +``` -The inventory API includes health check endpoints for readiness and liveness probes. +To stop use: -#### Readyz -The readyz endpoint checks if the service is ready to handle requests. -```bash -curl http://localhost:8081/api/inventory/v1/readyz +```shell +make inventory-down ``` -#### Livez -The livez endpoint checks if the service is alive and functioning correctly. -```bash -curl http://localhost:8081/api/inventory/v1/livez +#### Kessel-Inventory + Kafka + +In order to use the kafka configuration, one has to run strimzi and zookeeper. +You can do this by running; + +```shell +make inventory-up-kafka ``` -### Add hosts to inventory -To add hosts to the inventory, use the following `curl` command: +Start Kessel Inventory and configuring it to connect to kafka: +```yaml +eventing: + eventer: kafka + kafka: + bootstrap-servers: "localhost:9092" + # Adapt as required + # security-protocol: "SASL_PLAINTEXT" + # sasl-mechanism: PLAIN +``` + +You can use our default config with kafka by running: -```bash -curl -H "Content-Type: application/json" --data "@data/host.json" http://localhost:8081/api/inventory/v1beta1/resources/rhel-hosts +```shell +INVENTORY_API_CONFIG="./kafka-inventory-api.yaml" make run ``` -Depending on the config file you're using, the curl command will require additional headers for authorization of the request. +- Refer to [kafka-inventory-api.yaml](./kafka-inventory-api.yaml) for additional configuration -### Adding a new relationship (k8s-policy is propagated to k8s-cluster) -To add a k8s-policy_ispropagatedto-k8s-cluster relationship you can use the following `curl` command: +Once started, you can watch the messages using [kcat](https://github.com/edenhill/kcat) (formerly known as kafkacat) +or by exec into the running container like this: + +```shell +source ./scripts/check_docker_podman.sh +KAFKA_CONTAINER_NAME=$(${DOCKER} ps | grep inventory-api-kafka | awk '{print $1}') +${DOCKER} exec -i -t ${KAFKA_CONTAINER_NAME} /bin/bash + +# Once in the container +./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kessel-inventory +``` -```bash -curl -H "Content-Type: application/json" --data "@data/k8spolicy_ispropagatedto_k8scluster.json" http://localhost:8081/api/inventory/v1beta1/resource-relationships/k8s-policy_is-propagated-to_k8s-cluster +Manually terminate Kessel inventory and then run the following to stop kafka: + +```shell +make inventory-down-kafka ``` -To update it, use the `PUT` verb as follows: +#### Kessel Inventory + Kessel Relations + Keycloak -```bash -curl -X PUT -H "Content-Type: application/json" --data "@data/k8spolicy_ispropagatedto_k8scluster.json" http://localhost:8081/api/inventory/v1beta1/resource-relationships/k8s-policy_is-propagated-to_k8s-cluster +Similar as above, but instead of running Kafka, this will configure inventory to use a Keycloak service for authentication. + +- Sets up a keycloak instance running at port 8084 with [myrealm](myrealm.json) config file. +- Set up a default service account with clientId: `test-svc`. Refer to [get-token](scripts/get-token.sh) to learn how to fetch a token. +- Refer to [sso-inventory-api.yaml](./sso-inventory-api.yaml) for additional configuration + +To start use: +```shell +make inventory-up-sso ``` -And finally, to delete it, use the `DELETE` verb, notice that the data file is different this time. We only need the reporter data to delete a relationship. +Once it has started, you will need to fetch a token and use it when making calls to the service. -```bash -curl -X DELETE -H "Content-Type: application/json" --data "@data/relationship_reporter_data.json" http://localhost:8081/api/inventory/v1beta1/resource-relationships/k8s-policy_is-propagated-to_k8s-cluster +To get a token use: +```shell +make get-token ``` -### Running with `make run` +You can then export an ENV with that value and use in calls such as: -We are using the included `.inventory-api.yaml` file which allows guest access. -Guest access currently [makes use](https://github.com/project-kessel/inventory-api/blob/main/internal/authn/guest/guest.go#L20) of the `user-agent` header to -populate the Identity header. +```shell +curl -H "Authorization: bearer ${TOKEN}" # ... +``` -[data/host.json](./data/host.json) uses the `reporter_id: user@example.com`, hence you will need the following command: +To stop use: -```bash -curl -H "Content-Type: application/json" --user-agent user@example.com --data "@data/host.json" http://localhost:8081/api/inventory/v1beta1/resources/rhel-hosts +```shell +make inventory-down-sso ``` -### Running with `make inventory-up` +#### Running in Ephemeral Cluster with Relations API using Bonfire + +Instructions to deploy Kessel Inventory in an ephemeral cluster can be found on [Kessel docs](https://cuddly-tribble-gq7r66v.pages.github.io/kessel/inventory-api/ephemeral/) + +### API/Proto files Changes + +Once there is any change in the `proto` files (under (/api/kessel)[./api/kessel]) an update is required. -This provides a [PSK file](https://github.com/project-kessel/inventory-api/blob/main/config/psks.yaml#L1) with a token "1234". -The default port in this setup are `8081` (http) and `9091`. +This command will generate code and an (openapi)[./openapi.yaml] file from the `proto files`. +```shell +make api +``` -The following command will add the host to the inventory: +We can run the following command to update if there are expected breaking changes. +```shell +make api_breaking +``` -```bash -curl -H "Content-Type: application/json" -H "Authorization: bearer 1234" --data "@data/host.json" http://localhost:8081/api/inventory/v1beta1/resources/rhel-hosts +### Build Container Images + +By default, the quay repository is `quay.io/cloudservices/kessel-inventory`. If you wish to use another for testing, set IMAGE value first +```shell +export IMAGE=your-quay-repo # if desired +make docker-build-push ``` -## Contribution -`make pr-check` +## Example Usage +All these examples use the REST API and assume we are running the default local version +adjustments needs to be made to the curl requests if running with different configuration, +such as port, authentication mechanisms, etc. -## Running Inventory api with sso (keycloak) docker compose setup -`make inventory-up-sso` +### Health check endpoints -* Set up a keycloak instance running at port 8084 with [myrealm](myrealm.json) -* Set up a default service account with clientId: `test-svc` and password. Refer [get-token](scripts/get-token.sh) -* Refer [sso-inventory-api.yaml](sso-inventory-api.yaml) for configuration -* Refer [docker-compose-sso.yaml](docker-compose-sso.yaml) for docker-compose +The Kessel Inventory includes health check endpoints for readiness and liveness probes. -Use service account user as `reporter_instance_id` +#### Readyz +The readyz endpoint checks if the service is ready to handle requests. +```shell +curl http://localhost:8000/api/inventory/v1/readyz ``` -"reporter_instance_id": "service-account-svc-test" + +#### Livez +The livez endpoint checks if the service is alive and functioning correctly. +```shell +curl http://localhost:8000/api/inventory/v1/livez ``` -Refer [host-service-account.json](data/host-service-account.json) -### Generate a sso token -`make get-token` +### Resource lifecycle -Export the token generated -`export TOKEN=` +Resources can be added, updated and deleted to our inventory. Right now we support the following resources: +- `rhel-host` +- `integration` +- `k8s-cluster` +- `k8s-policy` -Sample request with the authorization header +To add a rhel-host to the inventory, use the following `curl` command -`curl -H "Authorization: bearer ${TOKEN}" -H "Content-Type: application/json" --data "@data/host-service-account.json" http://localhost:8081/api/inventory/v1beta1/resources/rhel-hosts` +```shell +curl -H "Content-Type: application/json" --data "@data/host.json" http://localhost:8000/api/inventory/v1beta1/resources/rhel-hosts +``` -## Running Inventory api with kafka -Starts a local strimzi kafka and zookeeper: -```bash -make inventory-up-kafka +To update it: + +```shell +curl -XPUT -H "Content-Type: application/json" --data "@data/host.json" http://localhost:8000/api/inventory/v1beta1/resources/rhel-hosts ``` -Start `inventory-api` using the `./kafka-inventory-api.yaml` config. -```bash -./bin/inventory-api serve --config ./kafka-inventory-api.yaml +and finally, to delete it, note that we use a different file, as the only required information is the reporter data. + +```shell +curl -XDELETE -H "Content-Type: application/json" --data "@data/host-reporter.json" http://localhost:8000/api/inventory/v1beta1/resources/rhel-hosts ``` -In a separate terminal exec into the kafka pod so you can watch messages. -```bash -source ./scripts/check_docker_podman.sh -KAFKA_CONTAINER_NAME=$(${DOCKER} ps | grep inventory-api-kafka | awk '{print $1}') -${DOCKER} exec -i -t ${KAFKA_CONTAINER_NAME} /bin/bash +### Adding a new relationship (k8s-policy is propagated to k8s-cluster) + +To add a `k8s-policy_ispropagatedto-k8s-cluster` relationship, first lets add the related resources `k8s-policy` and `k8s-cluster`. + +```shell +curl -H "Content-Type: application/json" --data "@data/k8s-cluster.json" http://localhost:8000/api/inventory/v1beta1/resources/k8s-clusters +curl -H "Content-Type: application/json" --data "@data/k8s-policy.json" http://localhost:8000/api/inventory/v1beta1/resources/k8s-policies ``` -Start consuming messages in the pod. -```bash -./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kessel-inventory +And then you can create the relation: + +```shell +curl -H "Content-Type: application/json" --data "@data/k8spolicy_ispropagatedto_k8scluster.json" http://localhost:8000/api/inventory/v1beta1/resource-relationships/k8s-policy_is-propagated-to_k8s-cluster +``` + +To update it: + +```shell +curl -X PUT -H "Content-Type: application/json" --data "@data/k8spolicy_ispropagatedto_k8scluster.json" http://localhost:8000/api/inventory/v1beta1/resource-relationships/k8s-policy_is-propagated-to_k8s-cluster ``` -In a separate terminal, post a resource to `inventory-api`: -```bash -curl -H "Content-Type: application/json" -H "Authorization: bearer 1234" --data "@data/k8s-cluster.json" http://localhost:8081/api/inventory/v1beta1/resources/k8s-clusters +And finally, to delete it, notice that the data file is different this time. We only need the reporter data. + +```shell +curl -X DELETE -H "Content-Type: application/json" --data "@data/relationship_reporter_data.json" http://localhost:8000/api/inventory/v1beta1/resource-relationships/k8s-policy_is-propagated-to_k8s-cluster ``` -Manually stop the `inventory-api` and then run `make inventory-down-kafka` -## Enable integration with Kessel relations API -Update the .inventory-api.yaml or inventory-api-compose.yaml +## Configuration -```yaml +### Enable integration with Kessel relations + +The default development config has this option disabled. You can check [Alternatives way of running this service](#alternatives-way-of-running-this-service) +for configurations that have Kessel relations enabled. +Supposing Kessel relations is running in `localhost:9000`, you can enable it by updating the config as follows: + +```yaml authz: impl: kessel kessel: @@ -175,7 +310,8 @@ authz: url: localhost:9000 enable-oidc-auth: false ``` -Enable oidc authentication with sso + +If you want to enable OIDC authentication with SSO, you can use this instead: ```yaml authz: @@ -184,37 +320,24 @@ authz: insecure-client: true url: localhost:9000 enable-oidc-auth: true - sa-client-id: "svc-test" + sa-client-id: "" sa-client-secret: "" sso-token-endpoint: "http://localhost:8084/realms/redhat-external/protocol/openid-connect/token" + ``` -## Running in Ephemeral Cluster with Relations API using Bonfire +## Testing -Deploy Relations API first with Bonfire following the steps available [HERE](https://cuddly-tribble-gq7r66v.pages.github.io/kessel/ephemeral/) +Tests can be run using: -Once its running, deploy Inventory using Bonfire: +```shell +make test +``` -`bonfire deploy kessel -C inventory-api --no-get-dependencies` +## Contributing -If you wish to test changes you've made that are unmerged, you can deploy them to ephemeral using a local config file -Note: this requires building the image first and pushing to your local quay (see make docker-build-push) +Follow the steps below to contribute: -```yaml -# example local config under $HOME/.config/bonfire/config -apps: -- name: kessel - components: - - name: inventory-api - host: local - repo: /path/to/inventory-api-repo - path: deploy/kessel-inventory.yaml - parameters: - INVENTORY_IMAGE: quay.io/your-repo/image-name - IMAGE_TAG: your-image-tag # latest is not recommended due to pull policy -``` - -Then run `bonfire deploy kessel -c $HOME/.config/bonfire/config.yaml --local-config-method override --no-get-dependencies` - -## Debugging Inventory API using Vscode -Follow the [DEBUG](./DEBUG.md) guide +- Fork the project +- Create a new branch for your feature +- Submit a pull request diff --git a/data/host-reporter.json b/data/host-reporter.json new file mode 100644 index 00000000..01d22449 --- /dev/null +++ b/data/host-reporter.json @@ -0,0 +1,9 @@ +{ + "reporter_data": { + "reporter_type": "OCM", + "reporter_version": "0.1", + "local_resource_id": "1", + "api_href": "www.example.com", + "console_href": "www.example.com" + } +} diff --git a/data/k8s-cluster.json b/data/k8s-cluster.json index 9c339103..8c44b749 100644 --- a/data/k8s-cluster.json +++ b/data/k8s-cluster.json @@ -7,7 +7,7 @@ "reporter_data": { "reporter_type": "ACM", "reporter_version": "0.1", - "local_resource_id": "1", + "local_resource_id": "2", "api_href": "www.example.com", "console_href": "www.example.com" }, diff --git a/docker-compose-sso.yaml b/docker-compose-sso.yaml index bdedf351..7f08258a 100644 --- a/docker-compose-sso.yaml +++ b/docker-compose-sso.yaml @@ -99,10 +99,9 @@ services: - "KEYCLOAK_ADMIN=admin" - "KEYCLOAK_ADMIN_PASSWORD=admin" volumes: - - ./myrealm.json:/opt/keycloak/data/import/myrealm.json + - ./myrealm.json:/opt/keycloak/data/import/myrealm.json:rw,z ports: - 8084:8084 - database: image: "postgres" command: -c track_commit_timestamp=on