Skip to content

Website translations documentation

Jump to bottom
BackNot edited this page Sep 19, 2024 · 2 revisions

OWASP SAMM Translations

The document is going to explain the whole process around the translation of the model YAML files, deploying the website with different translations, troubleshooting. It will start by giving brief overview of the whole process and then it will go into more technical details.

Overview

Translation is happening via Crowdin https://crowdin.com/project/owasp-samm

Each language has its own directory in the Crowdin project. Those YAML files are pushed from each language repository and when the translation is finished they are pushed back to the same repository.

Each translation is having its own Github repository (e.g. https://github.com/owaspsamm/i18n-FR/). Each repository has branches with the different versions of the model files (e.g trans_2.0). When the translations are ready and they are pushed to the branch, a Github Action can be run (from the Actions tab in the Github repository) which will convert the YAML files to markdown files and push them to "markdown" branch in the current language repository.

Later when the website 'deploy' CI/CD job is run Hugo (the website framework) will collect all translations (from the 'markdown' branches) from the different predefined repositories. Then the website will be deployed with the different languages.

image

Github ↔ Crowdin integration

We are using Github Crowdin integration (https://support.crowdin.com/github-integration/, https://support.crowdin.com/github-integration-sb/)

This requires that each translation has a branch with the source English files (e.g. src_2.0) and a branch with resulted translations (e.g. trans_2.0). The translated file will be pushed to the second branch (trans_2.0). Crowdin configuration (of the repository) happens in crowdin.yaml file which is telling what files to use and the resulting file names.

Crowdin will automatically open a pull requests in the repository to merge the resulted branch into the source branch (pull request is not mandatory to be accepted)

Language repositories

Each language should have it own repository (e.g. https://github.com/owaspsamm/i18n-FR/) This repository should contain a branch with the translated YAML files (e.g. trans_2.0). The original English files can be uploaded manually or a Github Action can be used to pull them.

Cloning original English model files to the language repository

This action is optional, but you will have to manually clone the English YAML model files when you want to start new translation version.

This is happening with shared workflow (https://docs.github.com/en/actions/sharing-automations/reusing-workflows) from the core repository. This is made so we don't duplicate the same logic across all language repositories (i.e. DRY).

# This is the workflow which will call the shared workflow which will clone the english model files

name: Get English files

# Controls when the workflow will run
on:
  workflow_dispatch:
     inputs:
      source_branch:
        description: 'The core repository branch to pull translation YAML files from (e.g. develop)'
        required: true
      source_folder:
        description: 'The folder in the core branch to pull translation YAML files from (e.g. model)'
        required: true
      new_branch_name:
        description: 'The name of the new branch to create in this repository with the pulled files (e.g. trans_2.0.8)'
        required: true

jobs:
  shared-workflow:
    uses: owaspsamm/core/.github/workflows/reusable-create-translation-branch.yml@438ef84c34d08befb5cd413a3344d1e6e1f75e9e
    with:
      source_branch: ${{inputs.source_branch}}
      source_folder: ${{inputs.source_folder}}
      new_branch_name: ${{inputs.new_branch_name}}

This action accepts 3 parameters. Source branch where the files are located, source folder where the files are located and the new branch name which will be created in the language repository.

owaspsamm/core/.github/workflows/eusable-create-translation-branch.yml@438ef84c34d08befb5cd413a3344d1e6e1f75e9e

This line will use reusable-create-translation-branch.yml workflow in owaspsamm/core repository. The last part is the commit hash of the most recent version of the shared workflow. If a new version of the workflow is released the commit hash have to be changed in each language repository so they use the newest version of the workflow.

image

Converting YAML files to markdown

This repository should contain Github Action that is going to be used to convert the YAML files to markdown files. All the language repositories are going to use a "shared workflow" file from the core repository (https://docs.github.com/en/actions/sharing-automations/reusing-workflows). This is made so we don't duplicate the same logic across all language repositories (i.e. DRY).

# This is example language action workflow file

name: Generate web markdown

# Controls when the workflow will run
on:
  workflow_dispatch:
    inputs:
      branch:
        description: 'Enter branch to use YAML files from'
        required: true
        type: string
      model_folder:
        description: 'Enter folder to use YAML files from'
        required: true
        type: string
jobs:
  shared-workflow-name:
    uses: owaspsamm/core/.github/workflows/reusable-yaml-process.yml@438ef84c34d08befb5cd413a3344d1e6e1f75e9e
    with:
      language: fr
      branch: ${{inputs.branch}}
      model_folder: ${{inputs.model_folder}}

This file will result in Github Action that is going to ask for two parameters.

The file presented is calling the shared workflow file in the core repository by this line:

 uses: owaspsamm/core/.github/workflows/reusable-yaml-process.yml@438ef84c34d08befb5cd413a3344d1e6e1f75e9e
    with:
      language: fr
      branch: ${{inputs.branch}}
      model_folder: ${{inputs.model_folder}}

This is calling

"owaspsamm" project,

"core" repository,

".github/workflows/reusable-yaml-process.yml" -- the shared workflow file

"438ef84c34d08befb5cd413a3344d1e6e1f75e9e" -- this is the commit hash of the file that is going to be used. If we update the "reusable-yaml-process.yml" file we must also update this commit hash in the language repository workflow file, otherwise an old version of the shared workflow will be used.

image

We are also passing the parameters that are entered when triggering the language repository workflow and another "language" parameter with the value "fr". This third parameter should be the language shortcode. Note that this parameter is going to be the URL prefix when the website is deployed (e.g. /fr/model/design)

After the Github Action is finished successfully the resulting files will be pushed on "markdown" branch in the language repository. Later when the website is deploying it will use those files.

Core model repository

Shared workflow file for converting YAML to markdown

The core model repository should contain the reusable workflow file that the language repositories are going to use to convert YAML files to markdown.

# File is reusable-yaml-process.yml
name: Reusable workflow example

on:
  workflow_call:
    inputs:
      language:
        required: true
        type: string
      branch:
        required: true
        type: string
      model_folder:
        required: true
        type: string
jobs:
  generate-markdown:
    runs-on: ubuntu-latest
    steps:
      - name: 'Checkout using release is workflow dispatched'
        uses: actions/checkout@v3
        with:
          ref: ${{ inputs.branch }}
      - name: 'Create output dir and copy files to override spaces in directories'
        run: |
          mkdir output
      - name: 'Generate model for website'
        uses: docker://backnot/owasp-samm-process-yaml-content:latest
        with:
          args: '-d ${{ inputs.model_folder }} -o output -l ${{ inputs.language }}'
      - name: 'Move generated files to common directory structure'
        run: |
          mkdir -p build/business-function/practice/stream
          BASE=output/markdown
          cp "$BASE"/{Design.md,Governance.md,Implementation.md,Operations.md,Verification.md} build/business-function
          cp "$BASE"/*-??-?.md build/business-function/practice/stream
          cp "$BASE"/*-??.md build/business-function/practice
      - name: Deploy
        uses: s0/git-publish-subdir-action@develop
        env:
          REPO: self
          BRANCH: markdown
          FOLDER: build
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          SQUASH_HISTORY: false

This file is going to accept three parameters -- language, model branch, model folder. It is going to use the process-yaml-content script (https://github.com/owaspsamm/process-yaml-content/ via Docker) that is used for converting the YAML files to markdown files.

Shared workflow file to clone English model files

This is the workflow that language repositories can call to clone the original model to new branches

name: Create new translation version branch

on:
  workflow_call:
    inputs:
      source_branch:
        required: true
        type: string
      source_folder:
        required: true
        type: string
      new_branch_name:
         required: true
         type: string

jobs:
  pull_and_create_branch:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout current repo
        uses: actions/checkout@v3
      - name: Set up Git with GitHub Actions bot identity
        run: |
          git config --global user.name "github-actions[bot]"
          git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
      - name: Pull content from another repository
        run: |
          git clone --branch ${{ inputs.source_branch }} --single-branch https://github.com/owaspsamm/core temp-repo
          rm -rf temp-repo/.git
          rsync -a --exclude='.git' temp-repo/ .
      - name: Create a new branch with the pulled content
        run: |
          git checkout -b ${{ inputs.new_branch_name }}
          git add ${{ inputs.source_folder }}
          git commit -m "Pulled content from ${{ inputs.source_repo }}:${{ inputs.source_branch }}"
          git push origin ${{ inputs.new_branch_name }}

Website

The website is using Hugo framework (https://gohugo.io/). In order for Hugo to recognize several languages there are two ways. The way it works now is that each language model files are in different folder (e.g. fr/es/). And the other layout files are just with different ending of names (e.g. model.md, model.fr.md, model.es.md - English, French, Spanish contents of the file model.md)

It is all happening in the Github Actions workflow file. In the file there are 2 jobs. "updateHugoMod" and "deploy".

This workflow can be triggered in three ways.

1. By pushing tag to core repository (it will signal to website repository by repository dispatch event)

2. By pushing to main branch

3. Manually from the "Github Actions" tab

The main difference between the three is that if it is triggered as in step 1 it will sync English model files from core repository new release branch tag.

In "updateHugoMod" job the process is going to download the model files from the different places - core repository for the English files and the language repositories for the others. Note the bold lines below - we are passing repository and branch to the job.

... 
- name: Get new release and commit to main
        if: github.event_name == 'repository_dispatch'
        env:
          RELEASE: "v${{ steps.semver_parser.outputs.fullversion }}"
        run: |
          echo "We got a new SAMM release: v${{ steps.semver_parser.outputs.fullversion }}${RELEASE}"
          hugo mod get "github.com/owaspsamm/core@markdown/${RELEASE}"
          cat go.mod 
      - name: Get translations
        run: |
          hugo mod get "github.com/owaspsamm/i18n-FR@markdown"
          hugo mod get "github.com/owaspsamm/i18n-ES@markdown"
...

In the website config/_default/config.toml file there is code that is importing/mounting the modules and telling the framework where each language files are located.

…...
[[module.imports]]
  # content
  path = "github.com/owaspsamm/core"
  disabled = false

  [[module.imports.mounts]]
  source = "business-function"
  target = "content/business-function"
  lang = "en"

[[module.imports]]
  # content
  path = "github.com/owaspsamm/i18n-FR"
  disabled = false

  [[module.imports.mounts]]
  source = "business-function"
  target = "content/business-function"
  lang = "fr"

[[module.imports]]
  # content
  path = "github.com/owaspsamm/i18n-ES"
  disabled = false

  [[module.imports.mounts]]
  source = "business-function"
  target = "content/business-function"
  lang = "es"
…...

languageCode = "en-us"
# Site language. Available translations in the theme's `/i18n` directory.
defaultContentLanguage = "en"
[languages]
  [languages.en]
    languageName = "English"
    weight = 1
    contentDir = "content/en"
  [languages.es]
    languageName = "Español"
    weight = 2
    contentDir = "content/es"
  [languages.fr]
    languageName = "French"
    weight = 3
    contentDir = "content/fr"

In the folder i18n there are yaml files with all the languages and some translated strings.

In layouts/shortcodes/language-dropdown.html this file will be used in the dropdown with all the available languages (shown at the model table).

How to ...

1.How to add new language?

  • Create new language repository

  • Add this as Github Actions workflow file inside the repository

name: Generate web markdown

# Controls when the workflow will run
on:
  workflow_dispatch:
    inputs:
      branch:
        description: 'Enter branch to use YAML files from'
        required: true
        type: string
      model_folder:
        description: 'Enter folder to use YAML files from'
        required: true
        type: string
jobs:
  shared-workflow-name:
    uses: owaspsamm/core/.github/workflows/reusable-yaml-process.yml@438ef84c34d08befb5cd413a3344d1e6e1f75e9e
    with:
      language: fr
      branch: ${{inputs.branch}}
      model_folder: ${{inputs.model_folder}}
  • Replace the hash after reusable-yaml-process.yml@ with the correct one (see it from core repository last commit of the file)

image

  • Add the language to the website config file (config/_default/config.toml)
[[module.imports]]
  # content
  path = "github.com/owaspsamm/i18n-FR"
  disabled = false

  [[module.imports.mounts]]
  source = "business-function"
  target = "content/business-function"
  lang = "fr"


….
[languages]
  [languages.en]
    languageName = "English"
    weight = 1
    contentDir = "content/en"
  [languages.es]
    languageName = "Español"
    weight = 2
    contentDir = "content/es"
  [languages.fr]
    languageName = "French"
    weight = 3
    contentDir = "content/fr"
  • Add the language repository to the Github Actions workflow YML file inside "updateHugoMod" "Get translations" step
 - name: Get translations
        run: |
          hugo mod get "github.com/owaspsamm/i18n-FR@markdown"
  • Add new file to i18n folder of the website

2.How to remove language?

  • If you want to just don't show the language in the website language dropdown you can remove the file for the corresponding language from i18n folder.
  • If you want full removal of the language from the website, reverse all the steps made to website repository from the above question
  • Also the corresponding repository should be removed from go.mod file of the website main branch

a1

3.How to convert language YAML files to markdown files?

  • Create branch named "markdown" in the language repository

  • Assuming that you have the required Github Action workflow file in the language repository (as specified in the above question regarding adding new language) you can trigger it from the Github Actions tab.

Clone this wiki locally