Skip to content

Quick start

Jump to bottom
Emilien Lancelot edited this page Sep 30, 2022 · 17 revisions

Quickstart

In this tutorial we'll go over the installation of Gitfaas and its functionnalities.
To do so we'll use the "replace_string" demo function found inside /demo/replace_string of this repository. It contains all the files you need to deploy a simple lambda.
This lambda will replace a word by another one inside a source string like the str.replace(...) of python would.

Installation

Cloning this repository

$ git clone https://github.com/rememberSoftwares/gitfaas.git
$ cd gitfaas/helm_chart

Install using Helm
You need:

  • a Git server URL
  • a Git username
  • a personnal token for authentificating against the Git server

More information on the git personal token : here

$ helm dep update
$ helm install gitfaas . -n gitfaas --create-namespace --set app.git.url="<YOUR_REPO_URL>" --set app.git.userName="<A_GIT_USERNAME>" --set app.git.http.personalToken="A_GIT_PERSONNAL_TOKEN"

⚠️ If your branch name isn't "main", add --set app.git.pullBranch="<YOUR_BRANCH_NAME>"

➡️ Check if Gitfaas container is happy: kubectl logs -n gitfaas gitfaas-XXXXXXX -c git -f. If there is no errors then the Git repo has been correctly cloned and Gitfaas is ready to apply whatever you need.

Commit the lambda's manifest

Move to /demo/replace_string/.

Check out the file /demo/replace_string/replace_job.yaml.

apiVersion: batch/v1
kind: Job
metadata:
  name: replace-job-{{RANDOM}}  # <== Replaced by a random string to provide always uniq Jobs (lambdas)
spec:
  ttlSecondsAfterFinished: 3600
  backoffLimit: 1
  template:
    spec:
      containers:
      - name: reverse
        image: wingardiumleviosa/demo-gitfaas-replace:0.0.1
        env:
        - name: PAYLOAD
          value: "{{PAYLOAD}}"        # <== Replaced by the message sent to this lambda
        - name: FUNCTION_UID
          value: "{{FUNCTION_UID}}"   # <== Replaced by a uniq UID used to post the lambdas response back to Gitfaas
      restartPolicy: Never

This yaml is the Kubernetes manifest corresponding to your lambda. It has three templating variables that will be replaced by Gitfaas during runtime.

Variable name Description
{{RANDOM}} A random string that you should add to the name of your job. This way each new instance of the job will have a unique name.
{{PAYLOAD}} This will be replaced by the content of the message sent to your lambda. Use it in ENV to access the value during runtime.
{{FUNCTION_UID}} This UID represents the unique run of the lambda. It can be used to return a response from inside the lambda by providing it to Gitfaas in addition to the response.

What this means, is that you can pass a message to your lambda and this message will be available inside the ENV variable PAYLOAD.
When the lambda has finished working it can send a response to the caller by using the request UID also available inside the ENV.

NB : The ttlSecondsAfterFinished: 3600 is a great Kubernetes functionnality. It means that the lambda will self destruct 3600 seconds (60 minutes) after it finishes. This way your namespace will not get polluted by tons of finished lambdas.

➡️ Commit this file to the Git repo linked to gitfaas

Configure a listening topic

Gitfaas waits for messages sent on topics to trigger the lambdas assign to this topic.
The topics and the assignes lamdas are defined in a really simple JSON formated configuration that must be commited into the Git repo.

/demo/replace_string/config.json

{
    "topics":[
        {
            "name": "replace",
            "configs": [
                "replace_job.yaml"
            ]
        }
    ]
}

We have defined a topic by its name inside of a list of topic. Here we created a "replace" topic.
To this topic we linked a file path which is "replace_job.yaml".
When Gitfaas gets a message on the topic "replace" it will apply the file "replace_job.yaml" on the cluster and forward it the message to the ENV (by replacing the templating variables {{PAYLOAD}}).

➡️ Commit this file at the root of the Git repo linked to Gitfaas.

Understanding the lambda's source code

⚠️ The job we commited earlier is using the image wingardiumleviosa/demo-gitfaas-replace:0.0.1. This image has been built using main.py and dockerfile. If you wish you can rebuild it and store it on your own image registry. Just remember to update the image inside the Job and to commit the update to Git.

This is the code of the lambda:
/demo/replace_string/main.py

import requests
import os
import base64
import json

# This function/lambda replaces a word inside a string. It takes 3 arguments.
# * A source message to work on.
# * A word that will be replaced
# * A word to replace with
#
# Ie: Expected payload format in input message
# {
#    "source": "I like bananas",
#    "str-to-replace": "bananas",
#    "replace-with": "apples"
# }

def main():

    # Retrieving the base64 payload (the input message sent to this function)
    b64_message = os.environ.get('PAYLOAD', None)
    # Retrieving the function UID (UID that identifies the current run of this function and allows to distinct multiple runs of the same function)
    function_uid = os.environ.get('FUNCTION_UID', None)

    if b64_message is None:
        print("No message in env")
        return

    # Decoding base64 JSON payload into usable JSON
    str_message = base64.b64decode(b64_message).decode("utf-8")
    json_message = json.loads(str_message)

    # Creating a new string using python's .replace() method. We replace the term contained in "str-to-replace" with the one from "replace-with"
    replaced_string = json_message["source"].replace(json_message["str-to-replace"], json_message["replace-with"])
    print("Replaced string is : %s" % replaced_string)

    # Returning a response to Gitfaas
    url = "http://gitfaas:5000/response/" + function_uid # We append our function_UID so that Gitfaas knows who is talking to him.

    try:
        # We send a response to Gitfaas containing the replaced string. Setting content type to text/plain or application/json is a good practice.
        ret = requests.post(url, data=replaced_string, headers={'Content-type': 'text/plain'})
        print("Response return = %s" % ret.text)

    except Exception as e:
        print("Error responding : %s" % str(e))


main()

In a nutshell : We retrieve the message sent to this lambda as a base64 string. The message is converted to JSON and the lambda does it's magic (it's just a replace inside a string). Then we post the result back to Gitfaas so the caller can access the result of the lambda's compute.

➡️ Let's break it appart from the top...

# Ie: Expected payload format in input message
# {
#    "source": "I like bananas",
#    "str-to-replace": "bananas",
#    "replace-with": "apples"
# }

Above is a simple comment. It tells users what this function expects as input message.
When we start our lambda, in order to do its work it must be given a payload similar to this to work properly or it will crash.

    b64_message = os.environ.get('PAYLOAD', None)
    function_uid = os.environ.get('FUNCTION_UID', None)

    if b64_message is None:
        print("No message in env")
        return

We start by retrieving from the ENV the message sent to the lambda. It's in the ENV variable PAYLOAD. We also retrieve the function UID from FUNCTION_UID.
The payload/message is what the function will work on. It must respect the format defined in the above comment or it will crash (input checking is not the aim of this tutorial).

    str_message = base64.b64decode(b64_message).decode("utf-8")
    json_message = json.loads(str_message)

The message from the ENV is a base64 string.

⚠️ Remember that all data transfer in Gitfaas arrives as base64. This allows to send and receive any kind of data format but the receiver always has to decode the value before it can be used !
After the base64 decode we still have a string. Let's convert that to JSON using json.loads.
We now posess a python dictionnary containing our input message to work on !

    replaced_string = json_message["source"].replace(json_message["str-to-replace"], json_message["replace-with"])
    print("Replaced string is : %s" % replaced_string)

This is the heart of the lambda. It's here the work is done. We replace the word given in "str-to-replace" with the one from "replace-with". Finaly we print the newly generated string (replaced_string).
This will appear in the logs when we do kubernetes logs.

    url = "http://gitfaas:5000/response/" + function_uid

Now that we have our replaced string we want to return this information to the caller. In order to do so we must POST the reponse on the route /response and provide the function UID which is uniq to this lambda's run.

    try:
        ret = requests.post(url, data=replaced_string, headers={'Content-type': 'text/plain'})
        print("Response return = %s" % ret.text)

    except Exception as e:
        print("Error responding : %s" % str(e))

This is the last section. We upload the result of the lambda to Gitfaas.
The line requests.post(url, data=replaced_string, headers={'Content-type': 'text/plain'}) makes a POST request on "/response" and provides the request UID.
We also set the content-type header to text/plain as we send the raw string directly. If we had returned a JSON value we could have set application/json.
ℹ️ An important thing to know is that this response will be stored as base64. So you can send pretty much any kind of format you want without any loss !

Great ! This covers getting the input messag, working with its value and finally responding something to the caller.

Deploying the lambda using the API

In a different terminal let's port-forward Gitfaas. Don't expose Gitfaas to the internet. It's not a safe place outhere !

$ kubectl port-forward -n gitfaas svc/gitfaas 5000:5000

Now that everything is in place we can launch any number of lambdas and pass it various payloads.
But first let's make sure Gitfaas is in sync with the Git repo. It pulls the repo each 5 minutes but you can force the update:

$ curl http://127.0.0.1:5000/refresh

Wait a few seconds and carry on...

$ PAYLOAD=$(cat << EOF
{
  "source": "I like bananas",
  "str-to-replace": "bananas",
  "replace-with": "apples"
}
EOF
)

$  curl -X POST http://127.0.0.1:5000/publish/replace -d "$PAYLOAD" -H 'Content-Type: application/json'
{"applies":[{"error":false,"message":"Applied successfully","path":"apply2/mygitrepo/replace_job.yaml"}],
"error":false,"requestUid":"r-33291581-9310-4d00-ba1a-2a2c5b05e66b"}

The Job has been applied successfully. Check inside the Gitfaas namespace if the job has started correctly and if the logs are correct.

$ kubectl get pod -n gitfaas
replace-job-3145c895-020d-41c5-a128-fda88042cf87--1-f4q5m   0/1     ContainerCreating   0          1s

$ kubectl logs replace-job-3145c895-020d-41c5-a128-fda88042cf87--1-f4q5m -n gitfaas
Replaced string is : I like apples
Gitfaas responded with = {"error":false,"message":"Response stored correctly"}

🚀 Perfect !

We can use the request UID to ask Gitfaas if the corresponding lambda has posted any response.

  • If it hasn't you'll get a null value, meaning that a lambda has been launched but didn't responded yet.
  • If the lambda did respond then you'll get the response encoded to base64 (next section).

Let's look at the reponse stored inside Gitfaas.

$ curl "http://127.0.0.1:5000/response/r-33291581-9310-4d00-ba1a-2a2c5b05e66b" # <== Replace with the requestUid that you got from /publish
{"error":false,"responses":["SSBsaWtlIGFwcGxlcw=="]}

In the above snipet you must replace the request UID r-33291581-9310-4d00-ba1a-2a2c5b05e66b by the one you got in the reponse from/publish.
Gitfaas anwsers with an array containing the reponse from your lambdas in base64 !

Let's decode the string:

$ echo SSBsaWtlIGFwcGxlcw== | base64 -d
I like apples

Great you have a response ! 👍

Responses are stored inside an array. If you had multiple lambda's listening to the replace topic and thoses lambdas have been triggered then you would have multiple values inside that array.

Now try again with this input message:

$ PAYLOAD=$(cat << EOF
{
  "source": "Kubernetes rocks",
  "str-to-replace": "rocks",
  "replace-with": "is great"
}
EOF
)

ℹ️ If you have JQ installed locally you can extract the response from Gitfaas like this:

$ curl "http://127.0.0.1:5000/response/r-33291581-9310-4d00-ba1a-2a2c5b05e66b" | jq -r ."responses[0]" | base64 -d
Kubernetes is great

Customizing the deployment during startup

Final stage in this tutorial.
You can render the Job manifest dynamically. For instance we can dynamically update the key backoffLimit: 0 to any value.
backoffLimit is a Job option that works like a retry when the container fails. By setting it to "0" Kubernetes won't create another pod instance. But if set to "1" and the pod crashes Kubernetes will spawn a new one.

To try this out we will update the Job manifest with a custom template variable.

apiVersion: batch/v1
kind: Job
metadata:
  name: replace-{{RANDOM}}
spec:
  ttlSecondsAfterFinished: 3600
  backoffLimit: {{CUSTOM-VAR}}   # <== This value will be rendered dynamically
  template:
    spec:
      containers:
      - name: replace
        image: wingardiumleviosa/demo-gitfaas-replace:0.0.1
        command: ["python3", "crash.py"]  # <== We update the CMD to make the container crash
        env:
        - name: PALYLOAD
          value: "{{PAYLOAD}}"
      restartPolicy: Never

➡️ Commit this updated version

To force Gitfaas to pull your commit immediately you can do:

$ curl http://127.0.0.1:5000/refresh

To define the value of CUSTOM-VAR we must use query params on the POST request. You can set as many as you want.

$ curl -X POST "http://127.0.0.1:5000/publish/replace?CUSTOM-VAR=1"

The section /replace?CUSTOM-VAR=1 is what define the value of backoffLimit. It will have the value "1".

ℹ️ Note that we haven't define any request body with -d as the container will crash it's not usefull. You can trigger lambdas but don't necessarily have to give them an input message. It depends on what does your lambdas.

$ kubectl get pods -n gitfaas
replace-job-89c6afc9-381d-43af-965a-baa3341e7f46--1-fgdr9   0/1     Error     0          10s
replace-job-89c6afc9-381d-43af-965a-baa3341e7f46--1-zxj6j   0/1     Error     0          15s

We have two pods. The original one and the one that got retried by kubernetes.
If you check the value of backoffLimit inside the Job you will see it has been set to 1.

Let's test with 4 retries:

$ kubectl delete job -n gitfaas # Cleaning the old jobs

$ curl -X POST "http://127.0.0.1:5000/publish/replace?CUSTOM-VAR=4"

$ kubectl get pods -n gitfaas
replace-job-b4853364-f161-426f-9942-bfcdc7d188de--1-69np8   0/1     Error     0          2m5s
replace-job-b4853364-f161-426f-9942-bfcdc7d188de--1-gh2ht   0/1     Error     0          90s
replace-job-b4853364-f161-426f-9942-bfcdc7d188de--1-gzdgd   0/1     Error     0          2m
replace-job-b4853364-f161-426f-9942-bfcdc7d188de--1-xv9bs   0/1     Error     0          10s
replace-job-b4853364-f161-426f-9942-bfcdc7d188de--1-zmhtx   0/1     Error     0          110s

ℹ️ Kubernetes might take some time to boot the 4 retries so don't worry if you don't instantly get 4.

This concludes this quickstart tutorial !
Remember that Gitfaas can apply any kind of Kubernetes manifests (if allowed in RBAC) not only Jobs. The templating system allows you to tweak everything so have fun !

Clone this wiki locally