Gooddata productivity tools

.fossa.yml

@@ -0,0 +1,7 @@
+---
+version: 3
+project:
+  id: gooddata-productivity-tools
+
+telemetry:
+  scope: "off"

.github/workflows/python.yaml

@@ -0,0 +1,54 @@
+name: Python tests and linting
+
+on:
+  push:
+    branches: ["master"]
+  pull_request:
+    branches: ["master"]
+
+jobs:
+  validate-python:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: checkout
+      uses: actions/checkout@v3
+
+    - name: Set up Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.11'
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install tox
+
+    - name: Run lint
+      run: tox -e lint
+
+    - name: Run type
+      run: tox -e type
+
+  test-python:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: checkout
+      uses: actions/checkout@v3
+
+    - name: Set up Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.11'
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install tox
+    
+    - name: Set up fake AWS credentials
+      run: sh tests/fake_aws_creds.sh
+
+    - name: Run tests
+      run: tox -e 3.11

.github/workflows/release.yaml

@@ -0,0 +1,26 @@
+---
+name: Release
+
+on:
+  push:
+    branches:
+      - master
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+      - name: Bump version and push tag
+        id: tag_version
+        uses: mathieudutour/[email protected]
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+      - name: Create a GitHub release
+        uses: ncipollo/release-action@v1
+        with:
+          tag: ${{ steps.tag_version.outputs.new_tag }}
+          name: Release ${{ steps.tag_version.outputs.new_tag }}
+          body: ${{ steps.tag_version.outputs.changelog }}

.gitignore

@@ -0,0 +1,26 @@
+.idea
+.vscode
+.venv*
+venv/
+.secrets
+__pycache__
+*.egg-info
+.DS_Store
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/

CONTRIBUTING.md

@@ -0,0 +1,32 @@
+# Contributing
+We want to make contributing to this project as easy and transparent as possible, whether it's:
+
+- Reporting a bug
+- Discussing the current state of the code
+- Submitting a fix
+- Proposing new features
+
+## We Develop with GitHub
+We use GitHub to host code, to track issues and feature requests, as well as accept pull requests.
+
+## We Use [GitHub Flow](https://guides.github.com/introduction/flow/index.html), So All Code Changes Happen Through Pull Requests
+Pull requests are the best way to propose changes to the codebase (we use [GitHub Flow](https://guides.github.com/introduction/flow/index.html)):
+
+1. Fork the repo and create your branch from `master`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed behaviour, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. Issue a pull request.
+
+## Any contributions you make will be under the MIT Software License
+In short, when you submit code changes, your submissions are understood to be under the same [MIT License](http://choosealicense.com/licenses/mit/) that covers the project. Feel free to contact the maintainers if that's a concern.
+
+## Report using GitHub's [issues](https://github.com/briandk/transcriptase-atom/issues)
+We use GitHub issues. If you want to report a bug or propose new feature, [open a new issue]().
+
+## Use a Consistent Coding Style
+Before creating pull request run `black` and `ruff` on the codebase to ensure consistent style and run `mypy` to check for typing errors.
+
+## License
+By contributing, you agree that your contributions will be licensed under its MIT License.

LICENCE

@@ -0,0 +1,11 @@
+BSD License
+
+Copyright (c) 2023-2024, GoodData Corporation. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted, provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,176 @@
+# gooddata-productivity-tools
+This repository contains tools that help with GoodData Cloud/CN workspace management, user management, and backup/restore of workspaces.
+
+This section of the documentation contains information on how to set up the environment and relevant authentication files. At the end of the Tools section, there is more specific documentation for each tool. The steps mentioned here are shared between them.
+
+## Requirements
+Python 3.10+
+
+Depending on your environment, the statements can start either as
+```sh
+pip  
+pip3
+```
+```sh
+python
+python3
+```
+please use the one that works for you and refers to python 3.10+.
+
+The version can be checked by running
+```sh
+python -V
+```
+
+## Install
+In order to install tooling requirements to the target environment, run the following: 
+
+```sh
+pip install -r requirements.txt
+```
+
+## Authentication
+
+Overall, the scripts developed within the repository follow the credential/authentication provisioning conventions of the GoodData and any used storage provider (e.g. AWS).
+
+The following section describes what credentials need to be set up, where to find them, and what format they should follow. If you need help with how to edit files in your user home folder (~), you can also refer to [step by step authentication setup guide](docs/SETUPATUHENTICATION.md).
+
+
+### GoodData
+When authenticating against GoodData, you can either export the required credentials using environment variables, or provide a GoodData profiles file.
+
+For example, you can export the environment variables like so:
+
+```sh
+export GDC_AUTH_TOKEN="some_auth_token"
+export GDC_HOSTNAME="https://host.name.cloud.gooddata.com/"
+```
+
+or you can choose to provide a GoodData `profiles.yaml` file of the following format:
+
+```yaml
+default:
+  host: https://host.name.cloud.gooddata.com/
+  token: some_auth_token
+
+customer:
+  host: https://customer.hostname.cloud.gooddata.com/
+  token: other_auth_token
+```
+
+By default, a tool attempts to locate a GoodData profile file at `~/.gooddata/profiles.yaml`, but you can also choose to provide a custom path like so:
+
+```sh
+python scripts/restore.py <input.csv> <conf.yaml> -p path/to/profiles.yaml
+```
+
+You can define multiple GoodData profiles in a single profiles file. By default, the `default` profile is used, but you can choose different one to use. For example, if you want to tell a tool to use the `customer` profile defined in the example `profiles.yaml` above, you can do so like this:
+
+```sh
+python scripts/restore.py <input.csv> <conf.yaml> -p path/to/profiles.yaml --profile customer
+```
+
+In case of providing both ways of authentication to a tool, the environment variables takes precedence and the profiles config is ignored.
+
+### AWS
+
+When authenticating against AWS, the [conventions made by the boto3 library](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) are followed.
+
+From the tool user perspective that means following the points 3. to 8. from the [Configuring Credentials section](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+
+One example of how you can supply AWS credentials for tools, is by defining one or more AWS profiles in `~/.aws/credentials` file.
+
+```
+[default]
+aws_access_key_id = some_key_id
+aws_secret_access_key = some_access_key
+
+[services]
+aws_access_key_id = other_key_id
+aws_secret_access_key = other_access_key
+```
+
+If you want to specify the specific AWS credentials profile to be used, see the tool-specific documentation.
+
+## Tools
+
+- [Backup workspace](docs/BACKUP.md)
+- [Restore workspace](docs/RESTORE.md)
+- [Workspace permission management](docs/PERMISSION_MGMT.md)
+- [User management](docs/USER_MGMT.md)
+
+
+## Known MacOS issue SSL: CERTIFICATE_VERIFY_FAILED
+
+If you are getting the following message:
+
+`Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)'))`
+
+it is likely caused by Python and it occurs if you have installed Python directly from python.org.
+
+To mitigate, please install your SSL certificates in HD -> Applications -> Python -> Install Certificates.command.
+
+---
+
+## Development
+
+This section is aimed towards developers wanting to adjust / test the code. If you are regular user you can ignore following parts.
+
+### Setup
+To set up local development environment do the following:
+
+1. (optional) Set up a local python virtual environment:
+
+```sh
+    python -m venv venv
+    source venv/bin/activate
+```
+
+2. Install tool, dev, and test requirements:
+
+```sh
+pip install -r requirements.txt -r requirements-test.txt -r requirements-dev.txt
+```
+
+
+### Style checking, linting, and typing
+The codebase (both, scripts and tests) is style, lint, and type checked when the CI/CD pipeline runs.
+
+Linting and style-checking is done with help of `black` and `ruff`.
+
+Type checking is done using `mypy`.
+
+To run either of the mentioned tools locally, just call the tool with a target directory.
+
+```sh
+<black|ruff|mypy> <target_path>
+```
+
+ For example, in order to check the typing in the scripts, call the following from the repository's root directory:
+
+```sh
+mypy scripts
+```
+
+
+### Testing
+The tooling test suite makes use of some third party tools, such as `pytest`, `tox`, and `moto`.
+
+To run the test suite locally, ensure you have test and script requirements installed (see Setup step above) change working directory to repository's root and then call:
+
+```sh
+pytest .
+```
+
+
+### Tox
+To run the test suite, linters and type checks locally you can also use `tox`.
+
+To check everything at once, ensure youre in the repository's root directory and simply call:
+
+```sh
+tox
+```
+
+## Contributing
+If you want to contribute to the project, please read the [contributing guide](CONTRIBUTING.md).

configuration_local.yaml

@@ -0,0 +1,2 @@
+storage_type: local
+storage:

configuration_s3.yaml

@@ -0,0 +1,5 @@
+storage_type: s3
+storage:
+  bucket: gdc-ms-dev-eu
+  backup_path: subfolder1/subfolder2/backups
+  profile: services