Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Development Documentation on Git/GitHub #370

Merged
merged 7 commits into from
Dec 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion documentation/gitbook/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,9 @@
* [Useful commands](how-to-run/useful-commands.md)
* [Tips, tricks, FAQ](how-to-run/tips-tricks-faq.md)

## 🗜️ Development
## [Development](./development/README.md)

* [Git and GitHub Usage](./development/git-and-github-usage.md)
* [Guidelines for contributors](development/python-guidelines-for-developers.md)

## Deprecated pages
Expand Down
3 changes: 3 additions & 0 deletions documentation/gitbook/development/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Development

This section covers development/contribution guidelines, including tutorials on how to setup your environment and guides on how we use git/GitHub.
27 changes: 27 additions & 0 deletions documentation/gitbook/development/git-and-github-usage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Git and GitHub Usage

We now use a modified gitflow style workflow for working with git and GitHub. For a detailed overview of this topic please refer to [Atlassian's article on Gitflow workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow).


## New Features

New features should be developed in a new branch checked out from the `dev` branch and then merged back into the `dev` branch via a PR on GitHub when ready for review. These feature branches can be deleted after merging into `dev`, unless someone from operations requests that it be kept around. For example, operations may want to merge the feature into their operational branch to get new functionality in advance of a release. By convention feature branches should be prefixed with `feature/<GitHub issue>/`, I.e. `feature/99/cool-new-thing`. Feature branch should also include edits to the GitBook documentation that describe their changes.


## Hot Fixes

Hot fixes should be developed in a new branch checked out from the `main` branch and merged back into the `main` branch via a PR on GitHub when ready for review. After successfully merging into `main` the hot fix branch should then be merged into `dev`, making appropriate adjustments to stabilize the feature. The priority for hot fixes is to correct a major issue quickly, so it is okay to delay detailed testing/documentation until merging into `dev`. By convention hot fix branches should be prefixed with `hotfix/`, I.e. `hotfix/important-fix-to-something`, and then converted into a feature branch after merging into main. These do not have to include edits to the GitBook documentation, but if the hotfix conflicts with what is described in the GitBook documentation it's **strongly recommended**.


## Creating Releases

Periodically releases will be created by merging the `dev` branch into `main` via a PR on GitHub and creating a new release the `main` branch after merging. These PRs should avoid discussion of individual feature changes, those discussions should be reserved for and handled in the feature PRs. If there is a feature that poses a significant problem in the process of creating a new release those changes should be treated like a new feature. The main purpose of this PR is to:

1. Resolve merge conflicts generated by hot fixes,
2. Making minor edits to documentation to make it clearer or more cohesive, and
3. Updating the `NEWS.md` file with a summary of the changes included in the release.


## Operations

Operational work should be developed in a new branch checked out from the `main` branch if there are modifications needed to the `flepiMoP` codebase. Pre-released features can be merged directly into this operational branch from the corresponding feature branch as needed via a git merge or rebase not a GitHub PR. After the operational cycle is over, the operations branch **should not** be deleted, instead should be kept around for archival reasons. Operational work needs to move quickly and usually does not involve documenting or testing code and is therefore unsuitable for merging into `dev` or `main` directly. Instead potential features should be extracted from an operations branch into a feature branch using [git cherry-pick](https://git-scm.com/docs/git-cherry-pick) and then modified into an appropriates state for merging into `dev` like a feature branch. By convention operations branch names should be prefixed with `operations/`, I.e. `operations/flu-SMH-2023-24`.
Original file line number Diff line number Diff line change
Expand Up @@ -78,58 +78,3 @@ For those using a Mac or Linux system for development this command is also avail
```bash
cp -f bin/pre-commit .git/hooks/
```

#### Structure of the main classes

The code is structured so that each of the main classes **owns** a config segment, and only this class should parse and build the related object. To access this information, other classes first need to build the object.

{% hint style="warning" %}
Below, this page is still underconstruction
{% endhint %}

The main classes are:

* `Coordinates:` this is a light class that stores all the coordinates needed by every other class (e.g the time serie
* `Parameter`
* `Compartments`
* `Modifers`
* `Seeding`,
* `InitialConditions`
* a `writeDF`
* function to plot
* (TODO: detail pipeline internal API)

### Batch folder

Here are some notes useful to improve the batch submission:

Setup site wide Rprofile.

```
export R_PROFILE=$COVID_PATH/slurm_batch/Rprofile
```

> SLURM copies your environment variables by default. You don't need to tell it to set a variable on the command line for sbatch. Just set the variable in your environment before calling sbatch.

> There are two useful environment variables that SLURM sets up when you use job arrays:

> SLURM\_ARRAY\_JOB\_ID, specifies the array's master job ID number. SLURM\_ARRAY\_TASK\_ID, specifies the job array index number. https://help.rc.ufl.edu/doc/Using\_Variables\_in\_SLURM\_Jobs

SLURM does not support using variables in the #SBATCH lines within a job script (for example, #SBATCH -N=$REPS will NOT work). A very limited number of variables are available in the #SBATCH just as %j for JOB ID. However, values passed from the command line have precedence over values defined in the job script. and you could use variables in the command line. For example, you could set the job name and output/error files can be passed on the sbatch command line:

```
RUNTYPE='test'
RUNNUMBER=5
sbatch --job-name=$RUNTYPE.$RUNNUMBER.run --output=$RUNTYPE.$RUNUMBER.txt --export=A=$A,b=$b jobscript.sbatch
```

However note in this example, the output file doesn't have the job ID which is not available from the command line, only inside the sbatch shell script.

#### File descriptions

launch\_job.py and runner.py for non inference job

inference\_job.py launch a slurm or aws job, where it uses

* \`inference\_runner.sh\` and inference\_copy.sh for aws
* ;batch/inference\_job.run for slurm