Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Example workflows for client commands #362

Open
voetberg opened this issue Aug 7, 2024 · 1 comment
Open

Example workflows for client commands #362

voetberg opened this issue Aug 7, 2024 · 1 comment
Labels
user User documentation

Comments

@voetberg
Copy link
Contributor

voetberg commented Aug 7, 2024

Client documentation would be more useful if it included examples of how commands were meant to be used.
Examples could include:

  • Snippet showing reading attributes of a deterministic RSE and adding new DIDs and rules that use it
  • Interactions between rules and replicas (making a rule, listing the replicas that rule impacts)
@voetberg voetberg added the user User documentation label Aug 7, 2024
@voetberg
Copy link
Contributor Author

This information can sit in the class wide client documentation. Key operations for different pieces of client infra can sit as an example under descriptions of the class.

Example (RSEClient):

Interact with Rucio Storage Elements. 
Use an RSE to define an addressable storage location. 
An RSE can have any number of meta attributes which can be used when defining rules for replica creation and transfers. 
RSEs have an internal ID, but are generally identified by their name. 
Once RSEs are set up, unless they need a new protocol, they are generally not modified. They have a loose correlation to physical storage locations and hardware, so adding a new RSE implies a new physicial storage location or a paritioning of existing storage.

Most RSE write operations can only be completed by someone with admin privilages. 

Additional information about RSEs can be found here - https://rucio.cern.ch/documentation/started/concepts/rucio_storage_element

Example Workflows: 

+ Create a new Disk-based RSE in the United States used in rule creation. 
  .. code-block:: python
from rucio.client import Client

c = Client()

# Create the new RSE
c.add_rse(
		  rse="MY-NEW-RSE", 
		  deterministic=True, 
		  city="Geneva", 
		  country="United States", 
		  rse_type="DISK"
)

# Add a read/write posix protocol 
protocol = {
		  # Name of the protocol. Must be unique on the RSE. 
		  'scheme': "file",
		  # How to locate the RSE by address 
		  'hostname': "my_new_rse.host_institution.edu",
		  'port': 0,
		  # Absolute path used as a prefix to all added files
		  'prefix': "/rse_storage/", 
		  # Package used to define the transfer protocol operations. 
		  # View [Protocol reference material???] for additional details. 
		  'impl': 'rucio.rse.protocols.posix.Default',
		  'domains': {
		      'wan': {
		          'read': 1,
		          'write': 1,
		          'delete': 1,
		          'third_party_copy_read': 1,
		          'third_party_copy_write': 1,
		      },
		      'lan': {
		          'read': 1,
		          'write': 1,
		          'delete': 1,
		      }
		  }
}

c.add_protocol(rse="MY-NEW-RSE", params=protocol)
# RSE can now be used for any rule that fit its attributes (a deterministic disk RSE in the US.)
# Transfers are only between RSEs with the same protocol (in this case, posix.)
# At this point, you can then look up subscriptions that may create new rules that use this RSE, and then reevaluate them. 


  + Querying existing RSEs, to see how many copies are possible for a rule.
  .. code-block:: python
   >> rses = Client().list_rses(rse_expression="rse_type=DISK")
   >> len([r for r in rses])
   3 
   >> client().add_replication_rule(did=[{"scope": "my-scope", "name": "my-dataset"}], copies=2, rse_expression="rse_type=DISK")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
user User documentation
Projects
None yet
Development

No branches or pull requests

1 participant