Add more details in guideline for Auth Istio&Apisix

authN-authZ/auth-apisix/README.md

@@ -2,13 +2,17 @@
 
 Follow the steps to enable authentication and authorization of OPEA services using APISIX api gateway and Identity provider like keycloak
 
-## Prerequisites
+## Start ChatQnA service
 
-1. Make sure ChatQnA service is running and accessible locally
+Please refer to [GenAIExamples ChatQnA](https://github.com/opea-project/GenAIExamples/tree/main/ChatQnA/kubernetes/intel) to start `chatqna` megaservice.
 
-2. Run keycloak, setup a realm with OIDC based authentication and add users with passwords for each user.
+## Start Keycloak and configuration
 
-Steps to start keycloak from official keycloak helm chart
+In this step, we run keycloak, setup a realm with OIDC based authentication.
+
+In this case, we add a realm called `apisix` and add a user called `mary` with password. In the authentication step, only the user from `apisix` realm can access the chatQnA pipeline.
+
+Steps to start keycloak.
 
 ```sh
 # Prerequisite: Create a PersistentVolume of 9Gi for keycloak-postgress with RWO access (to persist updated keycloak configuration)
@@ -26,24 +30,67 @@ spec:
   accessModes:
     - ReadWriteOnce
 EOF
+```
 
-# install keycloak with helm by setting
+Then install keycloak.
+
+```bash
+# (Option 1) install keycloak with helm by setting, you can change the user and password to your customized setting
 helm install keycloak oci://registry-1.docker.io/bitnamicharts/keycloak --version 22.1.0 --set auth.adminUser=admin --set auth.adminPassword=admin
 
-# Access keycloak UI through service/keycloak to do the necessary configurations
+#(Option 2) install keycloak with kubectl and configuration file
+cd ..
+kubectl apply -f ./keycloak_install.yaml
+```
+
+Get the ip and port to access keycloak.
+
+```bash
+export HOST_IP=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}' | cut -d '/' -f3 | cut -d ':' -f1)
+export KEYCLOAK_PORT=$(kubectl get svc keycloak -o jsonpath='{.spec.ports[0].nodePort}')
+export KEYCLOAK_ADDR=${HOST_IP}:${KEYCLOAK_PORT}
 ```
 
-Once the keycloak pod is up and running, access the UI through the Keycloak's NodePort service and set the necessary configuration
+**Note:** Double check if the host ip captured is the correct ip.
+
+Access the Keycloak admin console through the `KEYCLOAK_ADDR` and use the username and password specified before to login. Then we configure the users, here we create a user named "mary" and assign "user" role to her.
+
+The user management is done via Keycloak and the configuration steps look like this:
+
+1. Create a new realm named `apisix` within Keycloak.
+
+2. Create a new client called `apisix`, set `Client authentication` to `On`, `Save` setting and go to `Credentials` page, copy `Client Secret` for later usage.
+
+3. From the left pane select the Realm roles and create a new role name as `user`.
+
+4. Create a new user name as `mary`, set passwords for her (set 'Temporary' to 'Off'). Select Role mapping on the top, assign the `user` role to `mary`.
+
+5. Turn off the all the 'Required actions' under the 'Authentication' section in Keycloak
+
+Then set some environment variables.
+
+```bash
+export USER='mary'
+export PASSWORD=<password>
+export KEYCLOAK_REALM='apisix'
+export KEYCLOAK_CLIENT_ID='apisix'
+export KEYCLOAK_CLIENT_SECRET=<keycloak client secret>
+```
 
-## Update values
+**Trouble Shooting: https required**
 
-Update the following values in values.yaml
+If you meet "https required" issue when you open the console, you can fix with the following steps:
 
-1. Update all the entries in oidc config pertaining to your identity provider
+```bash
+kubectl exec -it ${keycloak_pods_id} -- /bin/bash
+cd /opt/keycloak/bin/
+./kcadm.sh config credentials --server ${KEYCLOAK_ADDR} --realm master --user admin ## need to type in password set before
+./kcadm.sh update realms/master -s sslRequired=NONE --server ${KEYCLOAK_ADDR}
+```
 
-2. Update all the entries in API specific configs
+Then after open the console and create `apisix` realm, go to "Realm setting", set "Require SSL" to "None"
 
-## Install
+## Install Apisix
 
 ```sh
 # Install apisix api gateway and ingress controller [reference: https://apisix.apache.org/docs/apisix/installation-guide/]
@@ -53,9 +100,21 @@ helm install auth-apisix apisix/apisix -f values_apisix_gw.yaml --create-namespa
 
 # WAIT UNTIL apisix-ingress-controller POD IS READY by checking status with 'kubectl get -n auth-apisix pods'
 # The pod is ready when READY status shows 1/1
+```
+
+</br>
+Apisix helm chart provides configs to change the service type to other options like LoadBalancer (apisix.service.type) and externalTrafficPolicy to 'local'(apisix.service.externalTrafficPolicy). These can be added in values_apisix_gw.yaml </br></br>
 
-# Publish authenticated APIs in APISIX gateway
-helm install auth-apisix-crds . --namespace auth-apisix
+### Publish authenticated APIs in APISIX gateway
+
+Check and update the oidc and megaservice config values in `apisix-chatqna-route.yaml` with environment variables, you can refer more in `templates` folder.
+
+For other OIDC Keycloak related information, you can get more at its discrovery endpoint `http://${KEYCLOAK_ADDR}/realms/apisix/.well-known/openid-configuration`
+
+Then apply the authentication for ChatQnA megaservice.
+
+```bash
+kubectl apply -f apisix-chatqna-route.yaml
 ```
 
 ## Usage
@@ -67,46 +126,31 @@ export NODE_PORT=$(kubectl get --namespace auth-apisix -o jsonpath="{.spec.ports
 export NODE_IP=$(kubectl get nodes --namespace auth-apisix -o jsonpath="{.items[0].status.addresses[0].address}")
 
 # the authenticated endpoint published in APISIX gateway can be accessed as: http://$NODE_IP:$NODE_PORT/<published endpoint uri>
-export accessUrl=http://$NODE_IP:$NODE_PORT/<your published endpoint uri>
-
-
+export accessUrl="http://$NODE_IP:$NODE_PORT/chatqna-oidc"
 ```
 
-</br>
-Apisix helm chart provides configs to change the service type to other options like LoadBalancer (apisix.service.type) and externalTrafficPolicy to 'local'(apisix.service.externalTrafficPolicy). These can be added in values_apisix_gw.yaml </br></br>
 While accessing the published APIs, the HTTP Authorization header of the request should contain the Access token provided by Identity provider as 'Bearer &ltAccess Token>'. </br></br>
 The access token, refresh token, userinfo, user roles and OIDC scopes assigned to user can be obtained by invoking OIDC auth endpoint through UI or token endpoint through curl and providing user credentials. </br></br>
 
 Below steps can be followed to get access token from keycloak and access the APISIX published ChatQnA API through curl
 
 ```sh
-# Get access token for specified user from keycloak
-export USER=<username>
-export PASSWORD=<password>
-export KEYCLOAK_ADDR=<keycloak url>
-export KEYCLOAK_REALM=<keycloak realm>
-export KEYCLOAK_CLIENT_ID=<keycloak client id>
-export KEYCLOAK_CLIENT_SECRET=<keycloak client secret>
-
 #Invoke Keycloak's OIDC token endpoint to get access token, refresh token and expirt times. (Only Access token is used in the example below)
 export TOKEN=$(curl -X POST http://${KEYCLOAK_ADDR}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token -H 'Content-Type: application/x-www-form-urlencoded' -d "grant_type=password&client_id=${KEYCLOAK_CLIENT_ID}&client_secret=${KEYCLOAK_CLIENT_SECRET}&username=${USER}&password=${PASSWORD}" | jq -r .access_token)
 
-# follow instructions above to fetch the NODE_IP and NODE_PORT
-export accessUrl="http://$NODE_IP:$NODE_PORT/chatqna-oidc"
-
 # try without token. Shall get response: "Authorization required 401 error"
-curl -X POST $accessUrl -d '{"text":"What is the revenue of Nike in 2023?","parameters":{"max_new_tokens":17, "do_sample": true}}' -sS -H 'Content-Type: application/json' -w " %{http_code}\n"
+curl -X POST $accessUrl -d '{"messages": "What is the revenue of Nike in 2023?", "max_new_tokens":17, "do_sample": true}' -sS -H 'Content-Type: application/json' -w " %{http_code}\n"
 
 # try with token. Shall get the correct response from ChatQnA with http code 200
-curl -X POST $accessUrl -d '{"text":"What is the revenue of Nike in 2023?","parameters":{"max_new_tokens":17, "do_sample": true}}' -sS -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -w " %{http_code}\n"
+curl -X POST $accessUrl -d '{"messages": "What is the revenue of Nike in 2023?", "max_new_tokens":17, "do_sample": true}' -sS -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -w " %{http_code}\n"
 
 ```
 
 ## Uninstall
 
 ```sh
 # Uninstall apisix
-helm uninstall auth-apisix-crds --namespace auth-apisix
+kubectl delete -f apisix-chatqna-route.yaml
 helm uninstall auth-apisix --namespace auth-apisix
 ```
 

authN-authZ/auth-apisix/apisix-chatqna-route.yaml

@@ -0,0 +1,33 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+  name: chatqna-authenticated-endpoints
+  namespace: default
+spec:
+  http:
+  - name: chatqna-query
+    match:
+      paths:
+      - /chatqna-oidc
+    backends:
+    - serviceName: chatqna
+      servicePort: 8888
+    plugins:
+    - name: openid-connect
+      enable: true
+      config:
+        client_id: ${KEYCLOAK_CLIENT_ID}
+        client_secret: ${KEYCLOAK_CLIENT_SECRET}
+        discovery: http://${KEYCLOAK_ADDR}/realms/${KEYCLOAK_CLIENT_ID}/.well-known/openid-configuration
+        introspection_endpoint: http://${KEYCLOAK_ADDR}/realms/${KEYCLOAK_CLIENT_ID}/protocol/openid-connect/token/introspect
+        introspection_endpoint_auth_method: client_secret_basic
+        scope: openid profile email
+        bearer_only: true
+        realm: ${KEYCLOAK_REALM}
+    - name: proxy-rewrite
+      enable: true
+      config:
+        uri: "/v1/chatqna"

authN-authZ/auth-istio/README.md

@@ -12,23 +12,46 @@ Here we use the chatQnA pipeline as an example.
 
 ## Prerequisite
 
-Before composing an OPEA pipeline with authN & authZ, user need to install Istio to support this feature. Please follow the steps [here](https://istio.io/latest/docs/setup/install/istioctl/) for Istio installation.
+**Istio installation**
+
+Before composing an OPEA pipeline with authN & authZ, user need to install Istio to support this feature.
+
+```bash
+curl -L https://istio.io/downloadIstio | sh -
+cd istio-1.23.2 ## change to your downloaded version
+export PATH=$PWD/bin:$PATH
+istioctl install
+```
+
+You can refer to [Istio installation](https://istio.io/latest/docs/setup/install/istioctl/) for detail information.
 
 **Deploy chatQnA pipeline and enable Istio sidecar injection**
 
-```sh
+(Optional) Create a new namespace and you can enable istio sidecar injection for the namespace directly.
 
-# deploy ChatQnA pipeline. You can either leverage GMC or the ChatQnA helm chart.
+```sh
 kubectl create ns chatqa
-# here's the command to leverage GMC custom resource for ChatQnA deployment.
+kubectl label namespace chatqa istio-injection=enabled
+```
+
+Deploy ChatQnA pipeline. You can either leverage GMC, manifest or helm chart.
+
+```bash
+# Option 1: GMC
+# leverage GMC custom resource for ChatQnA deployment.
 kubectl apply -f $(pwd)/../../microservices-connector/config/samples/ChatQnA/chatQnA_dataprep_xeon.yaml
-# please refer the doc https://github.com/opea-project/GenAIInfra/tree/main/helm-charts/chatqna for deployment with helm chart.
-# and install under `chatqa` namespace
+
+# Option 2: helm chart
+# please refer the doc https://github.com/opea-project/GenAIInfra/tree/main/helm-charts/chatqna for deployment with helm chart and install under `chatqa` namespace
+
+# Option 3: manifest
+# Please refer to https://github.com/opea-project/GenAIExamples/tree/main/ChatQnA/kubernetes/intel for deployment with manifest and install under `chatqa` namespace
 
 # expose an environment variable to set the deployment method
-# which will affect the configuration we use for authentication and authorization
+# which will affect the configuration we use for authentication and authorization. If you use manifest, can also set to "helm-chart-based"
 export DEPLOY_METHOD=<your deploy method, valid values: "gmc-based" and "helm-chart-based">
 
+# If you didn't  enable istio sidecar injection for "chatqna" namespace, here we
 # patch the deployment to enable istio sidecar injection
 # for GMC based deployment, set the `deployment-name` to `router-service-deployment`
 # for helm chart based deployment, set the `deployment-name` to `chatqna`
@@ -128,7 +151,7 @@ In this sample, we are going to test with the scenario that only privileged user
 Install Keycloak in the kubernetes cluster for user management. **Note:** Replace the admin password as your own in the command.
 
 ```bash
-# make sure running under authN-authZ/auth-istio folder
+# make sure running under authN-authZ folder
 kubectl apply -f $(pwd)/keycloak_install.yaml
 
 # get the ip and port to access keycloak.
@@ -151,18 +174,63 @@ The user management is done via Keycloak and the configuration steps look like t
 
 4. Create a new user name as `mary` and another user as `bob`. Set passwords for both users (set 'Temporary' to 'Off'). Select Role mapping on the top, assign the `user` role to `mary` and assign the `viewer` role to `bob`.
 
-**Apply authentication and authorization policies to the pipeline endpoint based on OIDC provider**
+5. Turn off the all the 'Required actions' under the 'Authentication' section in Keycloak
 
-Use the commands to apply the authentication and authorization rules.
+**Trouble Shooting: https required**
+
+If you meet "https required" issue when you open the console, you can fix with the following steps:
+
+```bash
+kubectl exec -it ${keycloak_pods_id} -- /bin/bash
+cd /opt/keycloak/bin/
+./kcadm.sh config credentials --server ${KEYCLOAK_ADDR} --realm master --user admin ## need to type in password set before
+./kcadm.sh update realms/master -s sslRequired=NONE --server ${KEYCLOAK_ADDR}
+```
+
+Then after open the console and create `istio` realm, go to "Realm setting", set "Require SSL" to "None"
+
+**Export the router service through istio ingress gateway**
+
+For authentication safegard, we should add a gateway for the service. Here we the istio ingress gateway will be used to access the chatQnA service in different setups.
+
+First export the router service through istio ingress gateway.
 
 ```bash
-# export the router service through istio ingress gateway
 kubectl apply -f $(pwd)/$DEPLOY_METHOD/chatQnA_router_gateway.yaml
+```
+
+Determine the ingress IP and ports and expose them as environment variables.
+
+```bash
+export INGRESS_NAME=istio-ingressgateway
+export INGRESS_NS=istio-system
+
+#run the following to determine if your Kubernetes cluster is in an environment that supports external load balancers:
+kubectl get svc "$INGRESS_NAME" -n "$INGRESS_NS"
+
+# Case1: If your EXTERNAL-IP value is <none> (or perpetually <pending>), your environment does not provide an external load balancer for the ingress gateway.
+# set the INGRESS_HOST to your host ip,
+export INGRESS_HOST=${host_ip}
+# set the INGRESS_PORT to the istio-ingressgateway svc port
+export INGRESS_PORT=${gateway_svc_port}
+
+#Case2: If your environment support external load balancers
+export INGRESS_HOST=$(kubectl -n "$INGRESS_NS" get service "$INGRESS_NAME" -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+export INGRESS_PORT=$(kubectl -n "$INGRESS_NS" get service "$INGRESS_NAME" -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
+```
+
+You can refter to the [istio ingress gateway guide](https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/#determining-the-ingress-ip-and-ports) for more details about ingress gateway ip and ports.
+
+**Apply authentication and authorization policies to the pipeline endpoint based on OIDC provider**
+
+Use the commands to apply the authentication and authorization rules.
 
+```bash
 # 'envsubst' is used to substitute envs in yaml.
 # use 'sudo apt-get install gettext-base' to install envsubst if it does not exist on your machine
 # apply the authentication and authorization rule
 # these files will restrict user access with valid token (with valid issuer, username and realm role)
+export REALM=istio
 envsubst < $(pwd)/$DEPLOY_METHOD/chatQnA_authN_keycloak.yaml | kubectl -n chatqa apply -f -
 envsubst < $(pwd)/$DEPLOY_METHOD/chatQnA_authZ_keycloak.yaml | kubectl -n chatqa apply -f -
 ```
@@ -181,7 +249,7 @@ export TOKENA=$(curl -X POST 'http://${KEYCLOAK_ADDR}/realms/istio/protocol/open
 export TOKENB=$(curl -X POST 'http://${KEYCLOAK_ADDR}/realms/istio/protocol/openid-connect/token' -H 'Content-Type: application/x-www-form-urlencoded' --data-urlencode 'grant_type=password' --data-urlencode 'client_id=istio' --data-urlencode 'username=bob' -d 'password=${PASSWORD}' -d 'scope=openid' -d 'response_type=id_token' | jq -r .access_token)
 ```
 
-Access the istio ingress gateway to reach the chatQnA service with different tokens. Follow the istio guide [here](https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/#determining-the-ingress-ip-and-ports) to determine the ingress IP and ports.
+Access the istio ingress gateway to reach the chatQnA service with different tokens.
 
 ```bash
 # follow the guide above to fetch the $INGRESS_HOST and $INGRESS_PORT
@@ -210,7 +278,7 @@ We are using a similar scenario here that only privileged users can access our c
 Here we take Keycloak as the sample OIDC Provider to use in the example. Follow the steps to install and configure Keycloak.
 
 ```bash
-# make sure running under authN-authZ/auth-istio folder
+# make sure running under authN-authZ folder
 kubectl apply -f $(pwd)/keycloak_install.yaml
 
 # get the ip and port to access keycloak.
@@ -330,3 +398,15 @@ sudo sed -i '1i\127.0.0.1       chatqna-ui.com' /etc/hosts
 Open browser with address `"chatqna-ui.com:${INGRESS_PORT}"` if using GMC based deployment. Otherwise, open the browser with address `"chatqna-service.com:${INGRESS_PORT}"`.
 
 Login with user `bob` and its credentials shall return a 403 error. Login with user `mary` and its credentials shall able to access the ChatQnA service.
+
+## Uninstall Istio
+
+After testing, you can uninstall Istio with following commands:
+
+```bash
+istioctl uninstall --purge -y
+kubectl delete namespace istio-system
+kubectl label namespace chatqa istio-injection-
+```
+
+Can refer to [Istio uninstal](https://istio.io/latest/docs/setup/getting-started/#uninstall) for more information.