Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Organising your data (aka. groups) #364

Merged
merged 7 commits into from
Jul 2, 2021
Merged

Organising your data (aka. groups) #364

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a8fd15b
Update content
CasperWA Jun 28, 2021
662c806
Remove empty lines in code blocks
CasperWA Jun 28, 2021
f07a4ea
Update text according to #360
CasperWA Jun 28, 2021
da0fccc
Merge branch 'tutorial-2021-intro-week' into close_342_organising-data
ramirezfranciscof Jul 1, 2021
c9c5931
Apply suggestions from code review
CasperWA Jul 2, 2021
9c0fc1a
Update according to review by @mbercx
CasperWA Jul 2, 2021
0ecf920
Merge branch 'tutorial-2021-intro-week' into close_342_organising-data
mbercx Jul 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 34 additions & 61 deletions docs/sections/data/groups.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,14 @@
# Organising your data

In this section of the tutorial we will focus on how to organise and explore the data in an AiiDA database.
As in (**TODO FIX LINK**), we will be using the previously created database entries for this tutorial.
To follow the tutorial then, you can use the profile in to which you have previously imported this data, or you may wish to create a fresh profile and import the archive into that:
For this tutorial you will be loading and using the information of an external pre-existing database.
To follow the tutorial, you can use the profile into which you have previously imported this data or import the archive now.
To do so, you can copy-paste the following code into a terminal after activating your AiiDA virtual environment:

```{code-block} console

$ verdi quicksetup --profile data
$ verdi profile setdefault data
$ verdi import https://object.cscs.ch/v1/AUTH_b1d80408b3d340db9f03d373bbde5c1e/marvel-vms/tutorials/aiida_tutorial_2020_07_perovskites_v0.9.aiida

verdi quicksetup --profile data
verdi profile setdefault data
verdi archive import https://object.cscs.ch/v1/AUTH_b1d80408b3d340db9f03d373bbde5c1e/marvel-vms/tutorials/aiida_tutorial_2020_07_perovskites_v0.9.aiida
```

## How to group nodes
Expand All @@ -23,10 +22,9 @@ A typical use case is to store all nodes that share a common property in a singl

### Listing existing groups

Lets explore the groups already present in the imported archive:
Let's explore the groups already present in the imported archive, by executing the following command:

```{code-block} console

$ verdi group list -a -A
PK Label Type string User
---- --------------- ------------- ---------------
Expand All @@ -37,28 +35,26 @@ PK Label Type string User
5 GBRV_pbesol core.upf aiida@localhost
6 GBRV_lda core.upf aiida@localhost
7 20200705-071658 core.import aiida@localhost

```

The default table shows us four pieces of information:

* **PK**: The Primary Key of the group
* **Label**: The label by which the group can be identified
* **Type string**: This tells us what "sub-class" of group this is.
* **PK**: The Primary Key of the group.
* **Label**: The label by which the group can be identified.
* **Type string**: This tells us what type of group this is.
Type strings can be used to class certain types of data, for example here we have general groups (`core`), groups containing pseudopotentials (`core.upf`), and an auto-generated group containing the nodes we imported from the archive (`core.import`).
mbercx marked this conversation as resolved.
Show resolved Hide resolved
For advanced use, you can create your own group subclass plugins, with specialised methods.
For advanced use, you can create your own group type plugins, with specialised methods by sub-classing the general `Group` class.
* **User**: The email of the user that created this group.

:::{tip}

The `-a` and `-A` flags we used above ensure that groups for all type strings and users are shown respectively.
The `-a` and `-A` flags used above ensure that groups for *all* type strings and users are shown, respectively.

:::

We can then inspect the content of a group by its label (if it is unique) or the PK:
You can then inspect the content of a group by its label (if it is unique) or the PK:

```{code-block} console

$ verdi group show tutorial_pbesol
----------------- ----------------
Group label tutorial_pbesol
Expand All @@ -71,54 +67,44 @@ PK Type Created
380 CalcJobNode 2078D:17h:46m ago
1273 CalcJobNode 2078D:18h:03m ago
...

```

Conversely, if you want to see all the groups a node belongs to, you can run:
Conversely, if you want to see all the groups a node belongs to, you can use its PK and run, (where `<PK>` should be the PK of the node):

```{code-block} console

$ verdi group list -a -A --node 380
$ verdi group list -a -A --node <PK>
PK Label Type string User
---- --------------- ------------- ---------------
1 tutorial_pbesol core aiida@localhost
7 20200705-071658 core.import aiida@localhost

```

### Creating and manipulating groups

Lets make a new group:
Let's make a new group:

```{code-block} console

$ verdi group create a_group
Success: Group created with PK = 8 and name 'a_group'

```

If we want to change the name of the group at any time:
You can change the name of a group using the `verdi group relabel` command:

```{code-block} console

$ verdi group relabel a_group my_group
Success: Label changed to my_group

```

Now we can add one or more nodes to it:
Add one or more nodes to your new group using node PKs from the `tutorial_pbesol` group we inspected earlier:

```{code-block} console

$ verdi group add-nodes -G my_group 380 1273
Do you really want to add 2 nodes to Group<my_group>? [y/N]: y

```

We can also copy the nodes from an existing group to another group:
You can also copy *all* nodes from an existing group to another group using:

```{code-block} console

$ verdi group copy tutorial_pbesol my_group
Warning: Destination group<my_group> already exists and is not empty.
Do you wish to continue anyway? [y/N]: y
Expand All @@ -135,31 +121,26 @@ PK Type Created
74 CalcJobNode 2078D:17h:51m ago
76 CalcJobNode 2078D:17h:57m ago
...

```

To remove nodes from the group run:
Removing nodes from the group is similar to adding them, try:

```{code-block} console

$ verdi group remove-nodes -G my_group 74
Do you really want to remove 1 nodes from Group<my_group>? [y/N]: y

```

and finally to remove the group entirely:

```{code-block} console

$ verdi group delete --clear my_group
Are you sure to delete Group<my_group>? [y/N]: y
Success: Group<my_group> deleted.

```

:::{important}

Any deletion operation related to groups won't affect the nodes themselves.
Any deletion operation related to groups will not affect the nodes themselves.
For example if you delete a group, the nodes that belonged to the group will remain in the database.
The same happens if you remove nodes from the group -- they will remain in the database but won't belong to the group any more.

Expand All @@ -168,65 +149,57 @@ The same happens if you remove nodes from the group -- they will remain in the d
## Organising groups in hierarchies

Earlier we mentioned that groups are like files in folders on your filesystem.
As with folders and sub-folders then, as the amount of groups we have grows, we may also wish to structure our groups in a hierarchy.
Groups in AiiDA are inherently "flat", in that groups may only contain nodes and not other groups.
However it is possible to construct *virtual* group hierarchies based on delimited group labels, using the `grouppath` utility.
Similar to folders and sub-folders, you may wish to structure your groups in a hierarchy once the number of groups in your database grows larger.
Groups in AiiDA are inherently "flat", meaning groups may only contain nodes and not other groups.
However, it is possible to construct *virtual* group hierarchies based on delimited group labels, using the `grouppath` utility.

Like folder paths grouppath requires delimitation by `/` (forward slash) characters.
Lets copy and rename the three tutorial groups:
Like folder paths on Unix systems, `grouppath` requires delimitation by forward slash (`/`) characters.
Let's copy and rename the three tutorial groups:

```{code-block} console

$ verdi group copy tutorial_lda tutorial/lda/basic
$ verdi group copy tutorial_pbe tutorial/gga/pbe
$ verdi group copy tutorial_pbesol tutorial/gga/pbesol

verdi group copy tutorial_lda tutorial/lda/basic
verdi group copy tutorial_pbe tutorial/gga/pbe
verdi group copy tutorial_pbesol tutorial/gga/pbesol
```

We can now list the groups in a new way:
You can now list the groups in a new way:

```{code-block} console

$ verdi group path ls -l
Path Sub-Groups
--------------- ------------
tutorial 3
tutorial_lda 0
tutorial_pbe 0
tutorial_pbesol 0

```

:::{note}

In the terminal, paths that contain nodes are listed in bold
In the terminal, paths that contain nodes are listed in bold.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this true? I don't seem to notice this. 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't actually check it now, but the code has bold=True? https://github.com/aiidateam/aiida-core/blob/develop/aiida/cmdline/commands/cmd_group.py#L448

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, i'll double check it during my pass, thanks!


:::

You can see that the actual groups that we create do not show, only the initial part of the "path", and how many sub-groups that path contains.
We can then step into a path:
You can see that the actual groups we just created do not appear, only the initial part of the path (`tutorial`), and how many subgroups that path contains.
You can then step into a path:

```{code-block} console

$ verdi group path ls -l tutorial
Path Sub-Groups
------------ ------------
tutorial/gga 2
tutorial/lda 1

```

This feature is also particularly useful in the verdi shell:
This feature is also particularly useful in the `verdi shell`:

```{code-block} ipython

In [1]: from aiida.tools.groups import GroupPath
In [2]: for subpath in GroupPath("tutorial/gga").walk(return_virtual=False):
...: print(subpath.get_group())
...:
"tutorial/gga/pbesol" [type core], of user aiida@localhost
"tutorial/gga/pbe" [type core], of user aiida@localhost

```

:::{seealso}
Expand Down