Add infrastructure for testing environment

[nirs]

This PR adds the drenv tool and python package for creating minikube based
local or CI test environment.

The changes includes a regional DR environment with ocm, rook, minio and rbd
mirroring.

The goal of this change:

• having an easy and reliable way to set up a testing environment, to make it
  easy for new contributors.
• having easy to understand and maintain code

When this work is finished, it should replace the hack deployment scripts.

Some components installed by hack/ocm-minimube.sh are missing:

• multicloud-operators-foundation
  https://github.com/stolostron/multicloud-operators-foundation

• multicloud-operators-subscription
  https://github.com/stolostron/multicloud-operators-subscription

• governance-policy-propagator
  https://github.com/open-cluster-management-io/governance-policy-propagator/

I'm not sure if these are needed, which version is required, which is
the right repo to install them from, and what changes are needed for
ramen.

Building and installing ramen is not done yet since it depends on the missing
components.

The next step is adding environment for metro DR with external ceph storage.
This will probably require some changes in the rook scripts.

[ShyamsundarR]

Have not tested it yet... but code looks fine.

[BenamarMk]

Older version of minikube will complain that --extra-disk option does not exist. A new version of minikube is needed.

[BenamarMk]

And after that it failed with this:

2022-10-28 15:02:16,610 ERROR   [ex1] Cluster failed
Traceback (most recent call last):
  File "/home/user1/projects/github/ramen/test/drenv/__main__.py", line 120, in execute
    f.result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/user1/projects/github/ramen/test/drenv/__main__.py", line 147, in start_cluster
    run_script(script, name=profile["name"])
  File "/home/user1/projects/github/ramen/test/drenv/__main__.py", line 178, in run_script
    run(script["file"], *script["args"], name=name)
  File "/home/user1/projects/github/ramen/test/drenv/__main__.py", line 201, in run
    f"[{name}] Command {cmd} failed rc={p.returncode}\n"
RuntimeError: [ex1] Command ('example/start', 'ex1') failed rc=125

Last messages:
  /usr/bin/env: invalid option -- 'S'
  Try '/usr/bin/env --help' for more information.

[nirs]

I'll add a note about minimal minkube version.

Which OS are you running? this was tested on Fedora 36, which has:

$ env --version
env (GNU coreutils) 9.0

But we the -S argument is needed only for the using unbuffered output, and we can solve
this in a better way.

[BenamarMk]

I am using CentOS 7.
The patch you posted fixed the problem.

[nirs]

Using Centos 7 (8 years old now?) is a bit extreme for development tool, I think it will
be hard to support it.

I think supporting current Fedora (and maybe current Ubuntu/Debian) is good enough.

[nirs]

Older version of minikube will complain that --extra-disk option does not exist. A new version of minikube is needed.

Current version mention --extra-disk and the minikube version that I tested with.

[nirs]

@BenamarMk thanks for testing! I hope the last commit works for you.

[ShyamsundarR]

(future) Is it possible to configure per module args from the higher level input YAML file (like test/regional-dr.yaml). The intention being, we could run different versions of Rook or other dependent components across different test.

With the current scheme each modules args would need to be edited.

[nirs]

Good idea, I think we need to make this much more declarative as we go.

[nirs]

Broken now since cluster-manager (probably also klusterlet) installs now v0.9.1, but the code
is hardcoding v0.8.0. Working on a fix.

[nirs]

With latest commit we work now with latest cluster-manager and klusterlet, but we need to remove the hard coded 0.9.0 tags.

[nirs]

This yaml looks exactly like the alm example - can we drop it and use the example from
the csv?

$ minikube kubectl -p hub -- get csv -n operators cluster-manager.v0.9.1 -o jsonpath='{.metadata.annotations.alm-examples}' | jq
[
  {
    "apiVersion": "operator.open-cluster-management.io/v1",
    "kind": "ClusterManager",
    "metadata": {
      "name": "cluster-manager"
    },
    "spec": {
      "deployOption": {
        "mode": "Default"
      },
      "placementImagePullSpec": "quay.io/open-cluster-management/placement:v0.9.0",
      "registrationConfiguration": {
        "featureGates": [
          {
            "feature": "DefaultClusterSet",
            "mode": "Enable"
          }
        ]
      },
      "registrationImagePullSpec": "quay.io/open-cluster-management/registration:v0.9.0",
      "workImagePullSpec": "quay.io/open-cluster-management/work:v0.9.0"
    }
  },
  {
    "apiVersion": "operator.open-cluster-management.io/v1",
    "kind": "ClusterManager",
    "metadata": {
      "name": "cluster-manager"
    },
    "spec": {
      "deployOption": {
        "hosted": {
          "registrationWebhookConfiguration": {
            "address": "management-control-plane",
            "port": 30443
          },
          "workWebhookConfiguration": {
            "address": "management-control-plane",
            "port": 31443
          }
        },
        "mode": "Hosted"
      },
      "placementImagePullSpec": "quay.io/open-cluster-management/placement:v0.9.0",
      "registrationImagePullSpec": "quay.io/open-cluster-management/registration:v0.9.0",
      "workImagePullSpec": "quay.io/open-cluster-management/work:v0.9.0"
    }
  }
]

[nirs]

I think can be dropped and we can use the alm-example from the csv (see previous commit).

$ minikube kubectl -p dr1 -- get csv -n operators klusterlet.v0.9.1 -o jsonpath='{.metadata.annotations.alm-examples}' | jq
[
  {
    "apiVersion": "operator.open-cluster-management.io/v1",
    "kind": "Klusterlet",
    "metadata": {
      "name": "klusterlet"
    },
    "spec": {
      "clusterName": "cluster1",
      "deployOption": {
        "mode": "Default"
      },
      "externalServerURLs": [
        {
          "url": "https://localhost"
        }
      ],
      "namespace": "open-cluster-management-agent",
      "registrationConfiguration": {
        "featureGates": [
          {
            "feature": "AddonManagement",
            "mode": "Enable"
          }
        ]
      },
      "registrationImagePullSpec": "quay.io/open-cluster-management/registration:v0.9.0",
      "workImagePullSpec": "quay.io/open-cluster-management/work:v0.9.0"
    }
  }
]

[ShyamsundarR]

In both cases, yes it can be used instead of adding another yaml to our repo.

[BenamarMk]

LGTM

[nirs]

@ShyamsundarR do you want to merge the current ugly fixes as is? I can post a better version
but I can also post this as the next pr.