Skip to content

Basic Examples

Jump to bottom
Ben Murray edited this page Apr 21, 2021 · 11 revisions

Basic Examples

As of ExeTera version 0.5.0, the API and its usage has changed dramatically, with usability and familiarity to users of Pandas being our primary goal. These updated examples are focussed on getting you started with simple operations such as creating, opening and interacting with DataSets, DataFrames and Fields.

Sessions

Creating a session object

Creating a Session object can be done multiple ways, but we recommend that you wrap the session in a context manager (with statement). This allows the Session object to automatically manage the datasets that you have opened, closing them all once the with statement is exited. Opening and closing datasets is very fast. When working in jupyter notebooks or jupyter lab, please feel free to create a new Session object for each cell.

from exetera.core.session import Session

# recommended
with Session() as s:
  ...

# not recommended
s = Session()

Loading dataset(s)

Once you have a session, the next step is typically to open a dataset. Datasets can be opened in one of three modes:

  • read - the dataset can be read from but not written to
  • append - the dataset can be read from and written to
  • write - a new dataset is created (and will overwrite an existing dataset with the same name)
with Session() as s:
  ds1 = s.open_dataset('/path/to/my/first/dataset/a_dataset.hdf5', 'r', 'ds1')
  ds2 = s.open_dataset('/path/to/my/second/dataset/another_dataset.hdf5', 'r+', 'ds2')

Closing a dataset

Closing a dataset is done through Session.close_dataset, as follows

with Session() as s:
  ds1 = s.open_dataset('/path/to/dataset.hdf5', 'r', 'ds1')

  # do some work
  ...

  s.close_dataset('ds1')

DataSet

Datasets are the object that maps to a given ExeTera datastore. When you open a dataset, it is a DataSet object that you get back. This can in turn be used to create, modify and delete DataFrames in the datastore.

with Session() as s:
  ds = s.open_dataset('/path/to/dataset.hdf5', 'r', 'ds')

  # list the tables present in this dataset
  for k in ds.keys():
    print(k)

DataFrame

DataFrames are designed with Pandas users in mind and have been created to explicitly be as close to Pandas as possible, given the differences between Pandas and ExeTera and how they represent data under the hood. DataFrames have a rich API that allows you to add, access and remove fields, as well as operations that can be carried out across all of the fields in the data frame.

Basic DataFrame manipulation

ds = # get a dataset from somewhere
df = ds['a_dataframe']

df2 = s.create_dataframe('another_dataframe') # create an empty dataframe

ds['a_copy'] = df # copy a dataframe

Adding and removing fields

ds = # get a dataset from somewhere
df = ds.create_dataframe('a_dataframe')

# create a set of (empty) fields
i_f = df.create_indexed_string('an_indexed_string_field')
f_f = df.create_fixed_string('a_fixed_string_field', 10)
n_f = df.create_numeric('a_numeric_field', 'int32')
c_f = df.create_categorical('a_categorical_field', 'int8', {0: b'a', 1: b'b'})
t_f = df.create_timestamp('a_timestamp_field')

# move / copy fields for assignment
df2 = ds['another_dataframe']
df3 = ds['yet_another_dataframe']
df2['b'] = df2['a'] # rename a field by assigning it
print('a' in df2) # -> False
df2['c'] = df3['c'] # copy a field between datasets

Fields

Get a field

The field being loaded must represent a valid field (see Concepts)

Getting fields has now been simplified. You can still call session.get, but it is simpler to fetch it directly from a DataFrame.

f = ds['a_field']
print(len(f))

Load all of the data for a field

r = session.get(dataset['patient_id'])
values = r.data[:]

Create a field

patients = dataset['patients']
timestamp = datetime.now(timezone.utc)
isf = session.create_indexed_string(patients, 'foo')
fsf = session.create_fixed_string(patients, 'bar', 10)
csf = session.create_categorical(patients, 'boo', {'no':0, 'maybe':2, 'yes':1})
nsf = session.create_numeric(patients, 'far', 'uint32')
tsf = session.create_timestamp(patients, 'foobar')

Write to a field in chunks

for c in chunks_from_somewhere:
    field.write_part(c)
field.flush()

Write to a field in a single go

field.write(generate_data_from_somewhere())

Referring to fields

Most of the session functions accept various representations of fields. The ones that are a bit more restrictive will be made more flexible in future releases. The following calls to apply_index are equivalent.

index = index_from_somewhere()
raw_foo = a_numpy_array_from_somewhere()
result = session.apply_index(index, src['foo']) # refer to the hdf5 Group that represents the field
result = session.apply_index(index, session.get(src['foo']) # refer to the field
result = session.apply_index(index, session.get(src['foo'].data[:]) # refer to the data in the field
result = session.apply_index(index, raw_foo) # refer to a numpy array
Clone this wiki locally