From 65e38eeea8dd0565fcaa75093547c005df0a63d5 Mon Sep 17 00:00:00 2001 From: Anurag Rajawat Date: Tue, 22 Oct 2024 14:40:44 +0530 Subject: [PATCH] docs: Update user-facing docs Signed-off-by: Anurag Rajawat --- README.md | 21 +- contribution/README.md | 80 ------- contribution/vagrant/.gitignore | 1 - contribution/vagrant/Vagrantfile | 51 ----- contribution/vagrant/env-setup.sh | 130 ----------- .../install-scripts/install-vagrant.sh | 30 --- .../install-scripts/install-virtualbox.sh | 24 -- deployments/sentryflow.yaml | 2 +- docs/CONTRIBUTING.md | 169 ++++++++++++++ docs/getting_started.md | 92 ++------ docs/receivers.md | 15 ++ .../ingress-controller/nginx-inc/nginx_inc.md | 216 ++++++++++++++++++ .../receivers/other/web-server/nginx/nginx.md | 95 ++++++++ docs/receivers/service-mesh/istio/istio.md | 56 +++++ 14 files changed, 585 insertions(+), 397 deletions(-) delete mode 100644 contribution/README.md delete mode 100644 contribution/vagrant/.gitignore delete mode 100644 contribution/vagrant/Vagrantfile delete mode 100755 contribution/vagrant/env-setup.sh delete mode 100755 contribution/vagrant/install-scripts/install-vagrant.sh delete mode 100755 contribution/vagrant/install-scripts/install-virtualbox.sh create mode 100644 docs/CONTRIBUTING.md create mode 100644 docs/receivers.md create mode 100644 docs/receivers/other/ingress-controller/nginx-inc/nginx_inc.md create mode 100644 docs/receivers/other/web-server/nginx/nginx.md create mode 100644 docs/receivers/service-mesh/istio/istio.md diff --git a/README.md b/README.md index 8e722b1..9d1dcff 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,7 @@ # SentryFlow -[![SentryFlow Docker Build](https://github.com/5gsec/sentryflow/actions/workflows/release.yml/badge.svg)](https://github.com/5gsec/sentryflow/actions/workflows/release.yml) [![CI for SentryFlow](https://github.com/5gsec/sentryflow/actions/workflows/ci-test-go.yml/badge.svg)](https://github.com/5gsec/sentryflow/actions/workflows/ci-test-go.yml) [![CI for AI Engine](https://github.com/5gsec/sentryflow/actions/workflows/ci-test-py.yml/badge.svg)](https://github.com/5gsec/sentryflow/actions/workflows/ci-test-py.yml) - -SentryFlow is a cloud-native system for API observability and security, specializing in log collection, metric production, and data exportation. +SentryFlow is a cloud-native system for API observability, specializing in log collection, and data +exportation. ## Architecture Overview @@ -10,17 +9,13 @@ SentryFlow is a cloud-native system for API observability and security, speciali ### Features -- Generation of API Access Logs -- Production of API Metrics -- AI-driven API Classification (Inference) - -## Documentation +- API access observability -### Basic Information +[//]: # (- Production of API Metrics) -- [Getting Started](docs/getting_started.md) -- [Use Cases](examples/README.md) +[//]: # (- AI-driven API Classification (Inference)) -### Contribution +## Documentation -- [Contribution Guide](contribution/README.md) +- [Getting Started](docs/getting_started.md) +- [Contribution Guide](docs/CONTRIBUTING.md) diff --git a/contribution/README.md b/contribution/README.md deleted file mode 100644 index 44c01f7..0000000 --- a/contribution/README.md +++ /dev/null @@ -1,80 +0,0 @@ -# Contribution Guide - -SentryFlow operates within Istio on Kubernetes. This means project participants will need an Istio environment. - -To minimize the hassle of installing (uninstalling) Kubernetes and configuring Istio, we have prepared a Vagrantfile that initializes an Ubuntu VM with fully functional Kubernetes and Istio. - -## 1. Prerequisites - -The provided Vagrantfile is tested on the following environment (i.e., Vagrant with VirtualBox). - -- **[Vagrant](https://www.vagrantup.com/)** - v2.2.9 -- **[VirtualBox](https://www.virtualbox.org/)** - v6.1 - -## 2. Starting up a VM - -To proceed, execute the following command within the `contribution/` directory: - -```bash -$ vagrant up -Bringing machine 'sentryflow' up with 'virtualbox' provider... -==> sentryflow: Importing base box 'generic/ubuntu2204'... -==> sentryflow: Matching MAC address for NAT networking... -==> sentryflow: Checking if box 'generic/ubuntu2204' version '4.3.10' is up to date... -... - sentryflow: clusterrolebinding.rbac.authorization.k8s.io/calico-node created - sentryflow: clusterrolebinding.rbac.authorization.k8s.io/calico-cni-plugin created - sentryflow: daemonset.apps/calico-node created - sentryflow: deployment.apps/calico-kube-controllers created -``` - -This command will initiate the installation of the necessary development environment. The duration of this process may vary, primarily depending on the speed of your network connection, and could take several minutes to complete. - -## 3. Development and Code Quality - -### Development - -Once Vagrant has been initialized successfully, you can access the Kubernetes environment by following these steps: - -``` -$ vagrant ssh -``` - -The source code for SentryFlow will be located at `/home/vagrant/sentryflow` within the virtual environment, and this directory will also be synchronized with the current working directory on the host machine. - -After making modifications to the source code of SentryFlow, you can build the changes by navigating to the `sentryflow` directory and running the Makefile. - -``` -make build -``` - -Executing the Makefile will result in the construction of container images, each tagged as specified. - -### Code Quality - -To maintain a clean and secure code base for SentryFlow, we conduct several checks, including `gofmt` for code formatting, `golint` for code style and linting, and `gosec` for security scanning. - -To evaluate the quality of your code, navigate to the `sentryflow` directory and execute the following commands: - -``` -make golint # run golint checks -make gofmt # run gofmt checks -make gosec # run gosec checks -``` - -### Pull Request - -Once everything is correctly set up, you are ready to create a pull request. Please refer to our guidelines for submitting PRs. - -## 4. Cleaning Up - -If you have successfully made changes to SentryFlow and wish to clean up the created workspace, you can simply use the following command: - -``` -$ vagrant destroy - sentryflow: Are you sure you want to destroy the 'sentryflow' VM? [y/N] y -==> sentryflow: Forcing shutdown of VM... -==> sentryflow: Destroying VM and associated drives... -``` - -Executing the command will terminate and remove the VM that you were working on. diff --git a/contribution/vagrant/.gitignore b/contribution/vagrant/.gitignore deleted file mode 100644 index a977916..0000000 --- a/contribution/vagrant/.gitignore +++ /dev/null @@ -1 +0,0 @@ -.vagrant/ diff --git a/contribution/vagrant/Vagrantfile b/contribution/vagrant/Vagrantfile deleted file mode 100644 index 08f0eca..0000000 --- a/contribution/vagrant/Vagrantfile +++ /dev/null @@ -1,51 +0,0 @@ -Vagrant.require_version ">= 2.0.0" - -VM_NAME = "sentryflow" -IMG_NAME = "generic/ubuntu2204" - -NUM_OF_VCPUS = 4 -SIZE_OF_VMEM = 4096 - -## == ## - -# create ssh keys if needed -system(" - if [ #{ARGV[0]} = 'up' ]; then - if [ ! -f ~/.ssh/id_rsa ]; then - echo '~/.ssh/id_rsa keys does not exist.' - ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa - fi - fi -") - -## == ## - -Vagrant.configure("2") do |config| - # vagrant@VM_NAME - config.vm.hostname = VM_NAME - - config.vm.define VM_NAME do |cfg| - cfg.vm.box = IMG_NAME - - cfg.vm.provider "virtualbox" do |vb| - vb.memory = SIZE_OF_VMEM - vb.cpus = NUM_OF_VCPUS - end - end - - # sync directories - config.vm.synced_folder "../../", "/home/vagrant/sentryflow", owner:"vagrant", group:"vagrant" - - # configure SSH - config.ssh.insert_key = false - - # copy ssh keys - config.vm.provision "file", source: "~/.ssh/id_rsa.pub", destination: "/home/vagrant/.ssh/id_rsa.pub" - config.vm.provision :shell, :inline => "cat /home/vagrant/.ssh/id_rsa.pub >> /home/vagrant/.ssh/authorized_keys", run: "always" - - # copy git config - config.vm.provision :file, source: "~/.gitconfig", destination: "$HOME/.gitconfig" - - # setup env - config.vm.provision "shell", path: "env-setup.sh" -end diff --git a/contribution/vagrant/env-setup.sh b/contribution/vagrant/env-setup.sh deleted file mode 100755 index 4c9e348..0000000 --- a/contribution/vagrant/env-setup.sh +++ /dev/null @@ -1,130 +0,0 @@ -#!/bin/bash - -# == Build Essential == # - -# update repo -sudo apt-get update - -# install build-essential -sudo apt-get install -y build-essential - -# == Containerd == # - -# add GPG key -sudo apt-get install -y curl ca-certificates gnupg -sudo install -m 0755 -d /etc/apt/keyrings -curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg -sudo chmod a+r /etc/apt/keyrings/docker.gpg - -# add docker repository -echo \ - "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ - "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \ - sudo tee /etc/apt/sources.list.d/docker.list > /dev/null - -# update the docker repo -sudo apt-get update - -# install containerd -sudo apt-get install -y containerd.io - -# set up the default config file -sudo mkdir -p /etc/containerd -sudo containerd config default | sudo tee /etc/containerd/config.toml -sudo sed -i "s/SystemdCgroup = false/SystemdCgroup = true/g" /etc/containerd/config.toml -sudo systemctl restart containerd - -# # == Kubernetes == # - -# install k3s -curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" INSTALL_K3S_EXEC="--disable=traefik" sh - - -echo "wait for initialization" -sleep 15 - -runtime="15 minute" -endtime=$(date -ud "$runtime" +%s) - -while [[ $(date -u +%s) -le $endtime ]] -do - status=$(kubectl get pods -A -o jsonpath={.items[*].status.phase}) - [[ $(echo $status | grep -v Running | wc -l) -eq 0 ]] && break - echo "wait for initialization" - sleep 1 -done - -# make kubectl accessable for vagrant user -mkdir -p /home/vagrant/.kube -sudo cp /etc/rancher/k3s/k3s.yaml /home/vagrant/.kube/config -sudo chown -R vagrant:vagrant /home/vagrant/.kube -echo "export KUBECONFIG=/home/vagrant/.kube/config" | tee -a /home/vagrant/.bashrc -PATH=$PATH:/bin:/usr/bin:/usr/local/bin - -# == Istio == # - -# move to home -cd /home/vagrant - -# download istio -curl -L https://istio.io/downloadIstio | sh - - -# copy istioctl to /usr/local/bin -sudo cp /home/vagrant/istio-*/bin/istioctl /usr/local/bin - -# change permissions -sudo chown -R vagrant:vagrant /home/vagrant/istio-* - -# install istio -su - vagrant -c "istioctl install --set profile=default -y" - -# == Docker == # - -# install Docker -sudo apt-get install -y docker-ce && sleep 5 - -# configure daemon.json -sudo mkdir -p /etc/docker -cat <> /home/vagrant/.bashrc -echo "export GOPATH=\$HOME/go" >> /home/vagrant/.bashrc -echo "export GOROOT=/usr/local/go" >> /home/vagrant/.bashrc -echo "export PATH=\$PATH:/usr/local/go/bin:\$HOME/go/bin" >> /home/vagrant/.bashrc -echo >> /home/vagrant/.bashrc - -# create a directory for Go -mkdir -p /home/vagrant/go -chown -R vagrant:vagrant /home/vagrant/go diff --git a/contribution/vagrant/install-scripts/install-vagrant.sh b/contribution/vagrant/install-scripts/install-vagrant.sh deleted file mode 100755 index 899bf0b..0000000 --- a/contribution/vagrant/install-scripts/install-vagrant.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash - -. /etc/os-release - -if [ "$NAME" != "Ubuntu" ]; then - echo "This script is for Ubuntu." - exit -fi - -if [ ! -x "$(command -v vagrant)" ]; then - VAGRANT_VERSION=2.3.0 - - # install wget - sudo apt-get -y install wget - - # download vagrant package - wget https://releases.hashicorp.com/vagrant/$VAGRANT_VERSION/vagrant_$VAGRANT_VERSION-1_amd64.deb - - # install vagrant - sudo apt-get -y install ./vagrant_$VAGRANT_VERSION-1_amd64.deb - - # rm the vagrant package - rm vagrant_$VAGRANT_VERSION-1_amd64.deb - - # install vagrant plugins - vagrant plugin install vagrant-vbguest - vagrant plugin install vagrant-reload -else - echo "Found Vagrant, skipping the installation of Vagrant." -fi diff --git a/contribution/vagrant/install-scripts/install-virtualbox.sh b/contribution/vagrant/install-scripts/install-virtualbox.sh deleted file mode 100755 index 8e84ae4..0000000 --- a/contribution/vagrant/install-scripts/install-virtualbox.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/bin/bash - -. /etc/os-release - -if [ "$NAME" != "Ubuntu" ]; then - echo "This script is for Ubuntu." - exit -fi - -if [ ! -x "$(command -v vboxmanage)" ]; then - # install wget - sudo apt-get -y install wget - - # download oracle_vbox_2016.asc and register it to the system - wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg --dearmor --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg - - # install vbox - sudo apt-get update - sudo apt-get install virtualbox - - echo "Please reboot the machine." -else - echo "Found VirtualBox, skipping the installation of Virtualbox." -fi diff --git a/deployments/sentryflow.yaml b/deployments/sentryflow.yaml index 13ed851..9f8aad0 100644 --- a/deployments/sentryflow.yaml +++ b/deployments/sentryflow.yaml @@ -55,7 +55,7 @@ subjects: apiVersion: v1 kind: ConfigMap metadata: - name: config + name: sentryflow namespace: sentryflow labels: app.kubernetes.io/part-of: sentryflow diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md new file mode 100644 index 0000000..3e7462a --- /dev/null +++ b/docs/CONTRIBUTING.md @@ -0,0 +1,169 @@ +# Want to contribute? + +Great! We welcome contributions of all kinds, big or small! This includes bug reports, code fixes, documentation +improvements, and code examples. + +Before you dive in, please take a moment to read through this guide. + +# Reporting issue + +We use [GitHub](https://github.com/5GSEC/SentryFlow) to manage the issues. Please open +a [new issue](https://github.com/5GSEC/SentryFlow/issues/new/choose) directly there. + +# Getting Started + +## Setting Up Your Environment + +- Head over to [GitHub](https://github.com/5GSEC/SentryFlow) and fork the 5GSec SentryFlow repository. +- Clone your forked repository onto your local machine. + ```shell + git clone git@github.com:/SentryFlow.git + ``` + +## Install development tools + +You'll need these tools for a smooth development experience: + +- [Make](https://www.gnu.org/software/make/#download) +- [Go](https://go.dev/doc/install) SDK, version 1.23 or later +- Go IDE ([Goland](https://www.jetbrains.com/go/) / [VS Code](https://code.visualstudio.com/download)) +- Container tools ([Docker](https://www.docker.com/) / [Podman](https://podman.io/)) +- [Kubernetes cluster](https://kubernetes.io/docs/setup/) running version 1.28 or later. +- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) version 1.28 or later. + +# Contributing Code + +## Building Locally + +- Install development tools as [mentioned above](#setting-up-your-environment). + +- Build SentryFlow using: + ```shell + cd sentryflow + make build + ``` + +## Understanding the Project + +Before contributing to any Open Source project, it's important to have basic understanding of what the project is about. +It is advised to try out the project as an end user. + +## Project Structure + +These are general guidelines on how to organize source code in this repository. + +``` +github.com/5GSEC/SentryFlow + +├── client -> Log client code. +├── deployments -> Manifests or Helm charts for deployment on Kubernetes. +├── docs -> All Documentation. +│   └── receivers -> Receiver specifc integration documentaion. +│   ├── other +│   │   ├── ingress-controller +│   │   │   └── nginx-inc +│   │   └── web-server +│   │   └── nginx +│   └── service-mesh +│   └── istio +├── filter -> Receivers specific filters/modules to observe API calls from receivers. +├── protobuf +│   ├── golang -> Generated protobuf Go code. +│   ├── python -> Generated protobuf Python code. +├── scripts +├── sentryflow +│   ├── cmd -> Code for the actual binary. +│   ├── config +│   │   └── default.yaml -> Default configuration file. +│   ├── go.mod -> Go module file to track dependencies. +│   └── pkg -> pkg is a collection of utility packages used by the components without being specific to its internals. +│   ├── config -> Configuration initialization code. +│   ├── core -> SentryFlow core initialization code. +│   ├── exporter -> Exporter code. +│   ├── k8s -> Kubernetes client code. +│   ├── receiver -> Receiver code. +│   │   ├── receiver.go -> All receivers initialization code. +│   │   └── svcmesh -> ServiceMesh receivers code. +│   │   └── other -> Other receivers code. +│   └── util -> Utilities. +``` + +## Imports grouping + +This project follows the following pattern for grouping imports in Go files: + +* imports from standard library. +* imports from other projects. +* imports from `sentryflow` project. + +For example: + +```go +import ( +"context" +"fmt" + +"k8s.io/apimachinery/pkg/runtime" +"sigs.k8s.io/controller-runtime/pkg/client" + +"github.com/5GSEC/SentryFlow/pkg/config" +"github.com/5GSEC/SentryFlow/pkg/receiver" +"github.com/5GSEC/SentryFlow/pkg/util" +) +``` + +## Pull Requests and Code Reviews + +We use GitHub [pull requests](https://github.com/5GSEC/SentryFlow/pulls) for code contributions. All submissions, +including +those from project members, require review before merging. +We typically aim for two approvals per pull request, with reviews happening within a week or two. +Feel free to ping reviewers if you haven't received feedback within that timeframe. + +### Commit Messages + +We follow the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) specification for clear and +consistent commit messages. + +Please make sure you have added the **Signed-off-by:** footer in your git commit. In order to do it automatically, use +the **--signoff** flag: + +```shell +git commit --signoff +``` + +With this command, `git` would automatically add a footer by reading your name and email from your `.gitconfig` file. + +### Merging PRs + +**For maintainers:** Before merging a PR make sure the title is descriptive and follows +a [good commit message](https://www.conventionalcommits.org/en/v1.0.0/). + +Merge the PR by using `Squash and merge` option on GitHub. Avoid creating merge commits. After the merge make sure +referenced issues were closed. + +# Testing and Documentation + +Tests and documentation are not optional, make sure your pull requests include: + +- Tests that verify your changes and don't break existing functionality. +- Updated [documentation](../docs) reflecting your code changes. +- Reference information and any other relevant details. + +## Commands to run tests + +- Unit tests: + ```shell + make tests + ``` + +- Integration tests: + ```shell + make integration-test + ``` + +- End-to-end tests: + ```shell + make e2e-test + ``` + diff --git a/docs/getting_started.md b/docs/getting_started.md index 4945f37..22d393f 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -1,94 +1,52 @@ # Getting Started -This guide provides a step-by-step process for deploying SentryFlow on Istio, aimed at enhancing API observability and security. It includes detailed commands for each step along with their explanations. +This guide provides a step-by-step process for deploying SentryFlow in a Kubernetes environment, aimed at enhancing API +observability. It includes detailed commands for each step along with their explanations. -> **Note**: SentryFlow is currently in the early stages of development. Please be aware that the information provided here may become outdated or change without notice. +> **Note**: SentryFlow is currently in the early stages of development. Please be aware that the information provided +> here may become outdated or change without notice. ## 1. Prerequisites -SentryFlow functions within the Istio framework. Below is a table detailing the environments where SentryFlow has been successfully deployed and verified to be operational. - -|System Name|Version| -|--|--| -|Ubuntu|22.04, 20.04| -|[Istio](https://istio.io/latest/)|1.20.2| -|[Kubernetes](https://kubernetes.io/)|v1.27.1| - -> **Note**: For information on Kubernetes configurations, including Container Network Interface (CNI), Container Runtime Interface (CRI), and their respective runtime settings, please consult the [compatability matrix](k8s_compatibility.md). +- A Kubernetes cluster running version 1.28 or later. +- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) version 1.28 or later. ## 2. Deploying SentryFlow -SentryFlow can be deployed using `kubectl` command. The deployment can be accomplished with the following commands: +Configure SentryFlow receiver by following [this](receivers.md). Then deploy SentryFlow by following `kubectl` command: -``` -$ git clone https://github.com/5GSEC/sentryflow -$ cd sentryflow/ -$ kubectl create -f deployments/sentryflow.yaml -namespace/sentryflow created -serviceaccount/sa-sentryflow created -clusterrole.rbac.authorization.k8s.io/cr-sentryflow created -clusterrolebinding.rbac.authorization.k8s.io/rb-sentyflow created -deployment.apps/sentryflow created -service/sentryflow created +```shell +kubectl apply -f https://raw.githubusercontent.com/5GSEC/SentryFlow/refs/heads/main/deployments/sentryflow.yaml ``` -This process will create a namespace named `sentryflow` and will establish the necessary Kubernetes resources. - -> **Note**: SentryFlow will automatically modify Istio's `meshConfig` to configure `extensionProviders`, facilitating SentryFlow's API log collection. +This will create a namespace named `sentryflow` and will deploy the necessary Kubernetes resources. Then, check if SentryFlow is up and running by: -``` -$ kubectl get pods -n sentryflow -NAME READY STATUS RESTARTS AGE -sentryflow-cd95d79b4-9q7d7 1/1 Running 0 4m41s +```shell +$ kubectl -n sentryflow get pods +NAME READY STATUS RESTARTS AGE +sentryflow-cff887bbd-rljm7 1/1 Running 0 73s ``` ## 3. Deploying SentryFlow Clients -SentryFlow has now been established within the cluster. In addition, SentryFlow exports API logs and metrics through gRPC. For further details on how this data is transmitted, please consult the [SentryFlow Client Guide](sentryflow_client_guide.md). - -For testing purposes, two simple clients have been developed. +SentryFlow has now been deployed in the cluster. In addition, SentryFlow exports API access logs through `gRPC`. -- `log-client`: Simply log everything coming from SentryFlow service -- `mongo-client`: Stores every logs coming from SentryFlow service to a MongoDB service. +For testing purposes, a client has been developed. -These clients can be deployed into the cluster under namespace `sentryflow` by following the command: +- `log-client`: Simply logs everything on `STDOUT` coming from SentryFlow. -- `log-client` - ``` - $ kubectl create -f deployments/log-client.yaml - deployment.apps/log-client created - ``` +It can be deployed into the cluster under namespace `sentryflow` by following the command: -- `mongo-client` - ``` - $ kubectl create -f deployments/mongo-client.yaml - deployment.apps/mongodb created - service/mongodb created - deployment.apps/mongo-client created - ``` - -Then, check if those clients and MongoDB are properly up and running by: - -``` -$ kubectl get pods -n sentryflow -NAME READY STATUS RESTARTS AGE -log-client-6c8864655f-h2sdv 1/1 Running 0 5m28s -mongo-client-7cbf6b888f-vd69g 1/1 Running 0 5m28s -mongodb-6f5d9fc599-zwnxj 1/1 Running 0 5m28s -... +```shell +kubectl apply -f https://raw.githubusercontent.com/5GSEC/SentryFlow/refs/heads/main/deployments/sentryflow-client.yaml ``` -If you observe `log-client`, `mongo-client`, and `mongodb` running within the namespace, the setup has been completed successfully. +Then, check if it is up and running by: -## 3. Use Cases and Examples - -Up to this point, SentryFlow has been successfully integrated into the Istio service mesh and Kubernetes cluster. For additional details on use cases and examples, please consult the accompanying documentation. - -The links below are organized by their level of complexity, starting from basic and progressing to more complex. +```shell +kubectl get pods -n sentryflow +``` -- [Single HTTP Requests](../examples/httpbin/README.md) -- [RobotShop Demo Microservice](../examples/robotshop/README.md) -- [Nephio Free5gc Workload](../examples/nephio/free5gc/README.md) -- [Nephio OAI Workload](../examples/nephio/oai/README.md) +If you observe `log-client`, is running, the setup has been completed successfully. diff --git a/docs/receivers.md b/docs/receivers.md new file mode 100644 index 0000000..317fbf8 --- /dev/null +++ b/docs/receivers.md @@ -0,0 +1,15 @@ +# SentryFlow Receivers + +SentryFlow supports following receivers: + +## Kubernetes + +- [Istio sidecar](https://istio.io/latest/docs/setup/) service mesh. To integrate SentryFlow with it, refer + to [this](receivers/service-mesh/istio/istio.md). +- [Nginx Inc.](https://github.com/nginxinc/kubernetes-ingress/) ingress controller. To integrate SentryFlow with it, + refer to [this](receivers/other/ingress-controller/nginx-inc/nginx_inc.md). + +## Non-Kubernetes + +- [Nginx web server](https://github.com/nginx/nginx) running on Virtual Machine or Bare-Metal. To integrate SentryFlow + with it, refer to [this](receivers/other/web-server/nginx/nginx.md). diff --git a/docs/receivers/other/ingress-controller/nginx-inc/nginx_inc.md b/docs/receivers/other/ingress-controller/nginx-inc/nginx_inc.md new file mode 100644 index 0000000..ab6fe3c --- /dev/null +++ b/docs/receivers/other/ingress-controller/nginx-inc/nginx_inc.md @@ -0,0 +1,216 @@ +# Nginx Incorporation Ingress Controller + +## Description + +This guide provides a step-by-step process to integrate SentryFlow +with [Nginx Inc.](https://github.com/nginxinc/kubernetes-ingress/) Ingress Controller, aimed at enhancing API +observability. It includes detailed commands for each step along with their explanations. + +SentryFlow make use of following to provide visibility into API calls: + +- [Nginx njs](https://github.com/nginx/njs) module. +- [Njs filter](../../../../../filter/nginx). + +## Prerequisites + +- Nginx Inc. Ingress Controller. + Follow [this](https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/) to deploy it. + +## How to + +To Observe API calls of your workloads running served by Nginx inc. ingress controller in Kubernetes environment, follow the below +steps: + +1. Create the following configmap in the same namespace as ingress controller. + +```shell +cat < +data: + sentryflow.js: | + const DEFAULT_KEY = "sentryFlow"; + const ResStatusKey = ":status" + const MAX_BODY_SIZE = 1_000_000; // 1 MB + + function requestHandler(r, data, flags) { + // https://nginx.org/en/docs/njs/reference.html#r_sendbuffer + r.sendBuffer(data, flags); + + // https://nginx.org/en/docs/njs/reference.html#r_done + r.done(); + + let responseBody = "" + try { + responseBody = new TextDecoder("utf-8") + .decode(new Uint8Array(data)); + } catch (error) { + r.error(`failed to decode data, error: ${error}`) + // Do not return, process other info even without body. + } + + if (responseBody.length > MAX_BODY_SIZE) { + responseBody = "" + } + + let apiEvent = { + "metadata": { + // Divide by 1000 converts the timestamp from milliseconds to seconds. + "timestamp": Date.parse(r.variables.time_iso8601.split("+")[0]) / 1000, + "receiver_name": "nginx", + "receiver_version": ngx.version, + }, + "source": { + "ip": r.remoteAddress, + "port": r.variables.remote_port, + }, + "destination": { + "ip": r.variables.server_addr, + "port": r.variables.server_port, + }, + "request": { + "headers": {}, + "body": r.requestText || "", + }, + "response": { + "headers": {}, + "body": responseBody, + }, + "protocol": r.variables.server_protocol, + }; + + for (const header in r.headersIn) { + apiEvent.request.headers[header] = r.headersIn[header]; + } + + // https://nginx.org/en/docs/http/ngx_http_core_module.html#variables + apiEvent.request.headers[":scheme"] = r.variables.scheme + apiEvent.request.headers[":path"] = r.uri + apiEvent.request.headers[":method"] = r.variables.request_method + + // Number of bytes sent to a client, not counting the response header; this + // variable is compatible with the “%B” parameter of the mod_log_config Apache module. + apiEvent.request.headers["body_bytes_sent"] = r.variables.body_bytes_sent + + // Request length including request line, header, and request body. + apiEvent.request.headers["request_length"] = r.variables.request_length + + // Request processing time in seconds with a milliseconds resolution; + // Time elapsed since the first bytes were read from the client. + apiEvent.request.headers["request_time"] = r.variables.request_time + + // Query (args) in the request line. + apiEvent.request.headers["query"] = r.variables.query_string + + for (const header in r.headersOut) { + apiEvent.response.headers[header] = r.headersOut[header]; + } + apiEvent.response.headers[ResStatusKey] = r.variables.status + + // https://nginx.org/en/docs/njs/reference.html#ngx_shared + ngx.shared.apievents.set(DEFAULT_KEY, JSON.stringify(apiEvent)); + } + + async function dispatchHttpCall(r) { + try { + let apiEvent = ngx.shared.apievents.get(DEFAULT_KEY); + await r.subrequest("/sentryflow", { + method: "POST", body: apiEvent, detached: true + }) + } catch (error) { + r.error(`failed to dispatch HTTP call to SentryFlow, error: ${error}`) + return; + } finally { + ngx.shared.apievents.clear(); + } + + r.return(200, "OK"); + } + + export default {requestHandler, dispatchHttpCall}; +EOF +``` + +2. Add the following volume and volume-mount in ingress controller deployment: + +```yaml +... +volumes: + - name: sentryflow-nginx-inc + configMap: + name: sentryflow-nginx-inc +... +... +volumeMounts: + - mountPath: /etc/nginx/njs/sentryflow.js + name: sentryflow-nginx-inc + subPath: sentryflow.js +``` + +3. Update ingress controller configmap as follows: + +```yaml +... +data: + http-snippets: | + js_path "/etc/nginx/njs/"; + subrequest_output_buffer_size 8k; + js_shared_dict_zone zone=apievents:1M timeout=300s evict; + js_import main from sentryflow.js; + + location-snippets: | + js_body_filter main.requestHandler buffer_type=buffer; + mirror /mirror_request; + mirror_request_body on; + + server-snippets: | + location /mirror_request { + internal; + js_content main.dispatchHttpCall; + } + + location /sentryflow { + internal; + + # Update SentryFlow URL with path to ingest access logs if required. + proxy_pass http://sentryflow.sentryflow:8081/api/v1/events; + + proxy_method POST; + proxy_set_header accept "application/json"; + proxy_set_header Content-Type "application/json"; + } +``` + +4. Update the `.receivers` configuration in `sentryflow` [configmap](../../../../deployments/sentryflow.yaml) as + follows: + + ```yaml + filters: + server: + port: 8081 + + # Following is required for `nginx-inc-ingress-controller` receiver. + nginxIngress: + deploymentName: + configMapName: + sentryFlowNjsConfigMapName: + + receivers: + others: + - name: nginx-inc-ingress-controller # SentryFlow makes use of `name` to configure receivers. DON'T CHANGE IT. + namespace: # Kubernetes namespace in which you've deployed the ingress controller. + ... + ``` + +5. Deploy SentryFlow + + ```shell + kubectl apply -f https://raw.githubusercontent.com/5GSEC/SentryFlow/refs/heads/main/deployments/sentryflow.yaml + ``` + +6. Trigger API calls to generate traffic. + +7. Use SentryFlow [log client](../../../../client) to see the API Events. diff --git a/docs/receivers/other/web-server/nginx/nginx.md b/docs/receivers/other/web-server/nginx/nginx.md new file mode 100644 index 0000000..cc7d0f5 --- /dev/null +++ b/docs/receivers/other/web-server/nginx/nginx.md @@ -0,0 +1,95 @@ +# Nginx Web Server + +## Description + +This guide provides a step-by-step process to integrate SentryFlow +with [Nginx webserver](https://github.com/nginxinc/kubernetes-ingress/), aimed at enhancing API +observability. It includes detailed commands for each step along with their explanations. + +SentryFlow make use of following to provide visibility into API calls: + +- [Nginx njs](https://github.com/nginx/njs) module. +- [Njs filter](../../../../../filter/nginx). + +## Prerequisites + +- Nginx web server. +- [Nginx-njs-module](https://github.com/nginx/njs?tab=readme-ov-file#downloading-and-installing). + +## How to + +To Observe API calls of your application running on a virtual machine (VM) behind a Nginx web server, follow the below +steps: + +1. Copy [sentryflow.js](../../../../../filter/nginx/sentryflow.js) file to `/etc/nginx/njs/` directory as + `sentryflow.js`. +2. Edit `nginx.conf` file located in `/etc/nginx/` directory as follows: + +```nginx configuration +load_module /etc/nginx/modules/ngx_http_js_module.so; +... +http { + ... + subrequest_output_buffer_size 8k; + js_path "/etc/nginx/njs/"; + js_shared_dict_zone zone=apievents:1M timeout=60s evict; + js_import main from sentryflow.js; + ... + server { + location / { + js_body_filter main.requestHandler buffer_type=buffer; + mirror /mirror_request; + mirror_request_body on; + } + + location /mirror_request { + internal; + js_content main.dispatchHttpCall; + } + + location /sentryflow { + internal; + + # SentryFlow URL with path to ingest access logs. + proxy_pass http:///api/v1/events; + + proxy_method POST; + proxy_set_header accept "application/json"; + proxy_set_header Content-Type "application/json"; + } + ... + } +} +``` + +Here is the sample [nginx.conf](../../../../../filter/nginx/nginx.conf) file for reference. + +3. Reload `nginx`: + +```shell +$ sudo nginx -s reload +``` + +4. Update the `.receivers` configuration in `sentryflow` [configmap](../../../../deployments/sentryflow.yaml) as + follows: + + ```yaml + filters: + server: + port: 8081 + + receivers: + others: + - name: nginx-webserver # SentryFlow makes use of `name` to configure receivers. DON'T CHANGE IT. + ... + ``` + +5. Deploy SentryFlow + + ```shell + kubectl apply -f https://raw.githubusercontent.com/5GSEC/SentryFlow/refs/heads/main/deployments/sentryflow.yaml + ``` + +6. Trigger API calls to generate traffic. + +7. Use SentryFlow [log client](../../../../client) to see the API Events. diff --git a/docs/receivers/service-mesh/istio/istio.md b/docs/receivers/service-mesh/istio/istio.md new file mode 100644 index 0000000..b2eaf28 --- /dev/null +++ b/docs/receivers/service-mesh/istio/istio.md @@ -0,0 +1,56 @@ +# Istio Sidecar Service Mesh + +## Description + +This guide provides a step-by-step process to integrate SentryFlow with Istio, aimed at enhancing API observability. It +includes detailed commands for each step along with their explanations. + +SentryFlow make use of following to provide visibility into API calls: + +- [Envoy Wasm Filter](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/wasm_filter) +- [Istio Wasm Plugin](https://istio.io/latest/docs/reference/config/proxy_extensions/wasm-plugin/) +- [Istio EnvoyFilter](https://istio.io/latest/docs/reference/config/networking/envoy-filter/) + +## Prerequisites + +- Deploy Istio service mesh. Follow [this](https://istio.io/latest/docs/setup/install/) to deploy it if you've not + deployed. +- Enable the envoy proxy injection by labeling the namespace in which you'll deploy your workloads: + ```shell + kubectl label ns istio-injection=enabled + ``` + +## How to + +To Observe API calls of your workloads running on top of Istio Service Mesh in Kubernetes environment, follow the below +steps: + +1. Update the `.receivers` configuration in `sentryflow` [configmap](../../../../deployments/sentryflow.yaml) as + follows: + + ```yaml + filters: + server: + port: 8081 + + # Envoy filter is required for `istio-sidecar` service-mesh receiver. + # Leave it as it is unless you want to use your filter. + envoy: + uri: 5gsec/sentryflow-httpfilter:v0.1 + + receivers: + serviceMeshes: + - name: istio-sidecar # SentryFlow makes use of `name` to configure receivers. DON'T CHANGE IT. + namespace: istio-system # Kubernetes namespace in which you've deployed Istio. + ... + ``` + +2. Deploy SentryFlow + + ```shell + kubectl apply -f https://raw.githubusercontent.com/5GSEC/SentryFlow/refs/heads/main/deployments/sentryflow.yaml + ``` + +3. Trigger API calls to generate traffic. + +4. Use SentryFlow [log client](../../../../client) to see the API Events.