SSO Migration & Container Deployment (#62)

* Dockerfile

* Refactor: Black formatter

Also include mypy into requirements.txt file

* Remove support for configuration files & Update packages

All the required configuration is now being set as environment variables
Also, packages used are updated

* Include OIDC authentication

Include AuthenticationMiddleware via core_lib

* Configure callback to authenticate using an access token

This configures two modules: `remote_apparatus.py` for requesting an access token
via client credentials grant and `controller.py` to give the required credentials via
environment variables when the job is submitted

* Enable OpenShift container account to create folders into relmons/
for job submission

* Set environment variables into HTCondor config file

Reference: https://htcondor.readthedocs.io/en/latest/man-pages/condor_submit.html

* Bugfix: RelMon validation status

Fix a logic error on /api/update endpoint that impedes the batch job running into HTCondor to update the RelMon status

* Bugfix: Include some extra logging instruction to debug behaviour into HTCondor

* Bugfix: Use the updated version of remote_apparatus.py into HTCondor

Also, update how environment variables are sent to HTCondor into JDS configuration file

* Bugfix: Double quote for wrapping environment variables

* Bugfix: Repository version

Clone directly the specific desired branch

* Bugfix: Repair finished

Remove access token from log file

* Update Pylint workflow

Use Ubuntu 22.04 for the runner and update Python version to 3.11.4

* Update Pylint

Some features the old Pylint version used where removed in Python 3.11
For more details, please see: python/cpython#28618

* Modify Pylint score

* Pylint: Avoid to scan core_lib and venv packages

* Pylint & GitHub Actions

1. Update Pylint configuration (.pylintrc)
2. Raise the acceptance score to 9.75
3. Fix some linting issues: Encoding errors, broad exceptions, etc.
4. Increase the GitHub Action checkout version to 4
5. Disable linting for core_lib and venv packages

* core_lib & Repository version for remote execution

1. Update core_lib to use the version available in PdmVWebCore/master
2. Update `local/file_creator.py` to pull the remote code from `master` instead of `SSOMigrationV2`

.dockerignore

@@ -0,0 +1,19 @@
+# Ignore the following files for image building
+.github/
+logs/
+venv/
+frontend/node_modules/
+frontend/dist/
+
+# Append the config file via secret files
+# or environment files
+config.cfg
+
+# Ignore Python cache
+__pycache__/
+
+# Ignore Docker Compose files
+relmonservice.yaml
+
+relmonsvc.sh
+*.pid

.github/workflows/pylint.yml

@@ -8,26 +8,25 @@ on:
 
 jobs:
   build:
-
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     name: Get newest code and run pylint
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v2
+      uses: actions/checkout@v4
       with:
         submodules: true
-    - name: Set up Python 3.6.8
+    - name: Set up Python
       uses: actions/setup-python@v4
       with:
-        python-version: "3.6.8"
+        python-version: "3.11.4"
     - name: Install dependencies
       run: |
         python3 -m pip install --upgrade pip
         python3 -m pip install -r requirements.txt
     - name: Run pylint
-      # --fail-under=9.5 - fail if score is below 9.5
+      # --fail-under=9.75 - fail if score is below 9.75
       # --fail-on=E - fail if there were errors, regardless of the score
       # --reports=y - print a report at the end
       run: |
         python3 -m pylint --version
-        python3 -m pylint --fail-under=9.5 --fail-on=E --reports=y `find . -type f | grep .py$ | xargs`
+        python3 -m pylint --fail-under=9.75 --fail-on=E --reports=y `find . -type d \( -path ./venv -o -path ./core_lib \) -prune -o -type f | grep .py$ | xargs`

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "core_lib"]
+	path = core_lib
+	url = https://github.com/cms-PdmV/PdmVWebCore.git
+	branch = SSOMigration

.pylintrc

@@ -60,88 +60,9 @@ confidence=
 # --enable=similarities". If you want to run only the classes checker, but have
 # no Warning level messages displayed, use "--disable=all --enable=classes
 # --disable=W".
-disable=print-statement,
-        parameter-unpacking,
-        unpacking-in-except,
-        old-raise-syntax,
-        backtick,
-        long-suffix,
-        old-ne-operator,
-        old-octal-literal,
-        import-star-module-level,
-        non-ascii-bytes-literal,
-        raw-checker-failed,
-        bad-inline-option,
-        locally-disabled,
-        file-ignored,
-        suppressed-message,
-        useless-suppression,
-        deprecated-pragma,
-        use-symbolic-message-instead,
-        apply-builtin,
-        basestring-builtin,
-        buffer-builtin,
-        cmp-builtin,
-        coerce-builtin,
-        execfile-builtin,
-        file-builtin,
-        long-builtin,
-        raw_input-builtin,
-        reduce-builtin,
-        standarderror-builtin,
-        unicode-builtin,
-        xrange-builtin,
-        coerce-method,
-        delslice-method,
-        getslice-method,
-        setslice-method,
-        no-absolute-import,
-        old-division,
-        dict-iter-method,
-        dict-view-method,
-        next-method-called,
-        metaclass-assignment,
-        indexing-exception,
-        raising-string,
-        reload-builtin,
-        oct-method,
-        hex-method,
-        nonzero-method,
-        cmp-method,
-        input-builtin,
-        round-builtin,
-        intern-builtin,
-        unichr-builtin,
-        map-builtin-not-iterating,
-        zip-builtin-not-iterating,
-        range-builtin-not-iterating,
-        filter-builtin-not-iterating,
-        using-cmp-argument,
-        eq-without-hash,
-        div-method,
-        idiv-method,
-        rdiv-method,
-        exception-message-attribute,
-        invalid-str-codec,
-        sys-max-int,
-        bad-python3-import,
-        deprecated-string-function,
-        deprecated-str-translate-call,
-        deprecated-itertools-function,
-        deprecated-types-field,
-        next-method-defined,
-        dict-items-not-iterating,
-        dict-keys-not-iterating,
-        dict-values-not-iterating,
-        deprecated-operator-function,
-        deprecated-urllib-function,
-        xreadlines-attribute,
-        deprecated-sys-function,
-        exception-escape,
-        comprehension-escape,
-        raise-missing-from,
-        consider-using-with,
-        unspecified-encoding
+disable=consider-using-f-string,
+        broad-exception-caught,
+        consider-using-with
 
 # Enable the message, report, category or checker with the given id(s). You can
 # either give multiple identifier separated by comma (,) or put this option
@@ -317,13 +238,6 @@ max-line-length=100
 # Maximum number of lines in a module.
 max-module-lines=1000
 
-# List of optional constructs for which whitespace checking is disabled. `dict-
-# separator` is used to allow tabulation in dicts, etc.: {1  : 1,\n222: 2}.
-# `trailing-comma` allows a space between comma and closing bracket: (a, ).
-# `empty-line` allows space-only lines.
-no-space-check=trailing-comma,
-               dict-separator
-
 # Allow the body of a class to be on the same line as the declaration if body
 # contains single statement.
 single-line-class-stmt=no
@@ -564,9 +478,3 @@ known-standard-library=
 # Force import order to recognize a module as part of a third party library.
 known-third-party=enchant
 
-
-[EXCEPTIONS]
-
-# Exceptions that will emit a warning when being caught. Defaults to
-# "BaseException, Exception".
-overgeneral-exceptions=BaseException,

Dockerfile

@@ -0,0 +1,40 @@
+# Build web application bundle
+FROM node:16-buster-slim@sha256:1417528032837e47462ea8cfe983108b0152f989e95cba2ddfbe0f0ddc2dcfbd AS frontend
+
+WORKDIR /usr/app
+
+COPY frontend .
+
+RUN npm install
+RUN npm run build
+
+# Build dependencies
+FROM python:3.11.3-alpine3.18@sha256:caafba876f841774905f73df0fcaf7fe3f55aaf9cb48a9e369a41077f860d4a7 AS build
+
+WORKDIR /usr/app
+RUN python -m venv /usr/app/venv
+ENV PATH="/usr/app/venv/bin:$PATH"
+
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+# Create image for deployment
+FROM python:3.11.3-alpine3.18@sha256:caafba876f841774905f73df0fcaf7fe3f55aaf9cb48a9e369a41077f860d4a7 AS backend
+
+RUN addgroup -g 1001 pdmv && adduser --disabled-password -u 1001 -G pdmv pdmv
+
+RUN mkdir /usr/app && chown pdmv:pdmv /usr/app
+WORKDIR /usr/app
+
+COPY --chown=pdmv:pdmv . .
+RUN rm -rf frontend/*
+
+COPY --chown=pdmv:pdmv --from=frontend /usr/app/dist ./frontend/dist
+COPY --chown=pdmv:pdmv --from=build /usr/app/venv ./venv
+
+RUN chmod -R 707 /usr/app/relmons/
+
+USER 1001
+
+ENV PATH="/usr/app/venv/bin:$PATH"
+CMD [ "python", "main.py" ]

environment.py

@@ -0,0 +1,98 @@
+"""
+This module parses some configuration variables from
+the runtime environment to use them in different sections
+from this application
+
+Attributes:
+    CALLBACK_URL (str): This is the url for the endpoint that HTCondor
+        jobs use to update the status for the running RelMon job.
+        For example: "https://cms-pdmv.cern.ch/relmonservice/api/update"
+        For more details, please see the endpoint definition: /api/update.
+    SERVICE_URL (str): This is the url for RelMonService2 application.
+        For example: "https://cms-pdmv.cern.ch/relmonservice"
+        It is used to include the application's url into email notifications
+        and for request cookies/tokens for authenticating callback request.
+    REPORTS_URL (str): This is the url for RelMon report page. 
+        For example: "https://cms-pdmv.cern.ch/relmon"
+        This is a static web page that renders all outputs for the reports.
+    SUBMISSION_HOST (str): This is the server where this application will open an SSH session
+        to submit jobs through HTCondor. For example: "lxplus.cern.ch"
+    REMOTE_DIRECTORY (str): This is the folder (into AFS or EOS) that stores 
+        all the required bundle files to submit a HTCondor job.
+    SERVICE_ACCOUNT_USERNAME (str): Username to authenticate to `SUBMISSION_HOST`
+    SERVICE_ACCOUNT_PASSWORD (str): Password to authenticate to `SUBMISSION_HOST`
+    EMAIL_AUTH_REQUIRED (bool): If this environment variable is provided,
+        the email client will authenticate to the email server. By default it is false,
+        because this anonymous server does not require to authenticate.
+    WEB_LOCATION_PATH (str): This is the path (AFS or EOS)
+        where all RelMon reports are going to be stored. This is the path used by `REPORT_URL`
+        application to load the reports static files.
+    TICK_INTERNAL (int): Elapsed time in seconds to perform a tick, please see `controller.tick()`
+        for more details.
+    MONGO_DB_HOST (str): MongoDB host for opening a client session.
+    MONGO_DB_PORT (int): MongoDB port for opening a client session.
+    MONGO_DB_USER (str): MongoDB user to authenticate a new client session.
+    MONGO_DB_PASSWORD (str): MongoDB password to authenticate a new client session.
+    HOST (str): Flask listening hostname
+    PORT (int): Flask port
+    DEBUG (bool): Enables DEBUG mode for RelMonService2 application
+    ENABLE_AUTH_MIDDLEWARE (bool): Enables the AuthenticationMiddleware to parse JWT 
+        or enable the application to handle OIDC flow by itself.
+    SECRET_KEY (str): Flask secret key.
+    CLIENT_ID (str): Client ID related to RelMonService2 application
+        or the reverse proxy that provides authentication.
+    CALLBACK_CLIENT_ID (str): Client ID for CLI integration application.
+    CALLBACK_CLIENT_SECRET (str): Client secret for CLI integration application.
+"""
+import os
+import inspect
+
+# RelMonService2 application
+CALLBACK_URL: str = os.getenv("CALLBACK_URL", "")
+SERVICE_URL: str = os.getenv("SERVICE_URL", "")
+REPORTS_URL: str = os.getenv("REPORTS_URL", "")
+SUBMISSION_HOST: str = os.getenv("SUBMISSION_HOST", "")
+REMOTE_DIRECTORY: str = os.getenv("REMOTE_DIRECTORY", "")
+SERVICE_ACCOUNT_USERNAME: str = os.getenv("SERVICE_ACCOUNT_USERNAME", "")
+SERVICE_ACCOUNT_PASSWORD: str = os.getenv("SERVICE_ACCOUNT_PASSWORD", "")
+EMAIL_AUTH_REQUIRED: bool = bool(os.getenv("EMAIL_AUTH_REQUIRED"))
+WEB_LOCATION_PATH: str = os.getenv("WEB_LOCATION_PATH", "")
+TICK_INTERVAL: int = int(os.getenv("TICK_INTERVAL", "600"))
+
+# MongoDB database
+MONGO_DB_HOST: str = os.getenv("MONGO_DB_HOST", "")
+MONGO_DB_PORT: int = int(os.getenv("MONGO_DB_PORT", "27017"))
+MONGO_DB_USER: str = os.getenv("MONGO_DB_USER", "")
+MONGO_DB_PASSWORD: str = os.getenv("MONGO_DB_PASSWORD", "")
+
+# Flask web server
+HOST: str = os.getenv("HOST", "0.0.0.0")
+PORT: int = int(os.getenv("PORT", "8000"))
+DEBUG: bool = bool(os.getenv("DEBUG"))
+ENABLE_AUTH_MIDDLEWARE: bool = bool(os.getenv("ENABLE_AUTH_MIDDLEWARE"))
+
+# OAuth2 credentials
+SECRET_KEY: str = os.getenv("SECRET_KEY", "")
+CLIENT_ID: str = os.getenv("CLIENT_ID", "")
+CALLBACK_CLIENT_ID: str = os.getenv("CALLBACK_CLIENT_ID", "")
+CALLBACK_CLIENT_SECRET: str = os.getenv("CALLBACK_CLIENT_SECRET", "")
+
+# Check that all environment variables are provided
+missing_environment_variables: dict[str, str] = {
+    k: v
+    for k, v in globals().items()
+    if not k.startswith("__")
+    and not inspect.ismodule(v)
+    and not isinstance(v, bool)
+    and not v
+}
+
+if missing_environment_variables:
+    msg: str = (
+        "There are some environment variables "
+        "required to be set before running this application\n"
+        "Please set the following values via environment variables\n"
+        "For more details, please see the description available into `environment.py` module\n"
+        f"{list(missing_environment_variables.keys())}"
+    )
+    raise RuntimeError(msg)