Skip to content

Commit

Permalink
Merge branch 'AIDASoft:master' into schema_fix
Browse files Browse the repository at this point in the history
  • Loading branch information
@hegner
hegner authored Jun 12, 2024
2 parents fcb1a13 + 16a113c commit 462e88c
Show file tree
Hide file tree
Showing 26 changed files with 316 additions and 239 deletions.
25 changes: 13 additions & 12 deletions doc/templates.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,18 +26,19 @@ Note that some of the information below will only apply to either of these gener
Currently PODIO loads templates that are placed in [`<prefix>/python/templates`](/python/templates).
They are broadly split along the classes that are generated for each datatype or component from the EDM definition:

| template file(s) | content | generated file(s) |
|---------------------------------|---------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|
| `Component.h.jinja2` | Definition for each component | `[<package>/]<component-name>.h` |
| `Data.h.jinja2` | POD struct of each datatype (living in the POD layer) | `[<package>/]<datatype-name>Data.h` |
| `Obj.{h,cc}.jinja2` | `Obj` class for each datatype (living in the object layer) and managing resources | `[<package>/]<datatype-name>Obj.h`, `src/<datatype-name>Obj.cc` |
| `[Mutable]Object.{h,cc}.jinja2` | The user facing interfaces for each datatype (living in the user layer) | `[<package>/][Mutable]<datatype-name>.h`, `src/[Mutable]<datatype-name>.cc` |
| `Collection.{h,cc}.jinja2` | The user facing collection interface (living in the user layer) | `[<package>/]<datatype-name>Collection.h`, `src/<datatype-name>Collection.cc` |
| `CollectionData.{h,cc}.jinja2` | The classes managing the collection storage (not user facing!) | `[<package>/]<datatype-name>CollectionData.h`, `src/<datatype-name>CollectionData.cc` |
| `selection.xml.jinja2` | The `selection.xml` file that is necessary for generating a root dictionary for the generated datamodel | `src/selection.xml` |
| `SIOBlock.{h,cc}.jinja2` | The SIO blocks that are necessary for the SIO backend | `[<package>/]<datatype-name>SIOBlock.h`, `src/<datatype-name>SIOBlock.cc` |
| `MutableStruct.jl.jinja2` | The mutable struct definitions of components and datatypes for julia |`[<package>/]<datatype-name>Struct.jl`, `[<package>/]<component-name>Struct.jl` |
| `ParentModule.jl.jinja2` | The constructor and collection definitions of components and datatypes in the data model are contained within a single module named after the package-name |`[<package>/]<package-name>.jl` |
| template file(s) | content | generated file(s) |
|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|
| `Component.h.jinja2` | Definition for each component | `[<package>/]<component-name>.h` |
| `Data.h.jinja2` | POD struct of each datatype (living in the POD layer) | `[<package>/]<datatype-name>Data.h` |
| `Obj.{h,cc}.jinja2` | `Obj` class for each datatype (living in the object layer) and managing resources | `[<package>/]<datatype-name>Obj.h`, `src/<datatype-name>Obj.cc` |
| `[Mutable]Object.{h,cc}.jinja2` | The user facing interfaces for each datatype (living in the user layer) | `[<package>/][Mutable]<datatype-name>.h`, `src/[Mutable]<datatype-name>.cc` |
| `Collection.{h,cc}.jinja2` | The user facing collection interface (living in the user layer) | `[<package>/]<datatype-name>Collection.h`, `src/<datatype-name>Collection.cc` |
| `CollectionData.{h,cc}.jinja2` | The classes managing the collection storage (not user facing!) | `[<package>/]<datatype-name>CollectionData.h`, `src/<datatype-name>CollectionData.cc` |
| `datamodel.h.jinja2` | The *full datamodel header* that includes everything of a generated EDM (via including all generated `Collections`). | `[<package>]/<package>.h` |
| `selection.xml.jinja2` | The `selection.xml` file that is necessary for generating a root dictionary for the generated datamodel | `src/selection.xml` |
| `SIOBlock.{h,cc}.jinja2` | The SIO blocks that are necessary for the SIO backend | `[<package>/]<datatype-name>SIOBlock.h`, `src/<datatype-name>SIOBlock.cc` |
| `MutableStruct.jl.jinja2` | The mutable struct definitions of components and datatypes for julia | `[<package>/]<datatype-name>Struct.jl`, `[<package>/]<component-name>Struct.jl` |
| `ParentModule.jl.jinja2` | The constructor and collection definitions of components and datatypes in the data model are contained within a single module named after the package-name | `[<package>/]<package-name>.jl` |


The presence of a `[<package>]` subdirectory for the header files is controlled by the `includeSubfolder` option in the yaml definition file.
Expand Down
23 changes: 0 additions & 23 deletions include/podio/CollectionBranches.h
View file

This file was deleted.

19 changes: 19 additions & 0 deletions include/podio/GenericParameters.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

#include <algorithm>
#include <iostream>
#include <iterator>
#include <map>
#include <memory>
#include <mutex>
Expand Down Expand Up @@ -105,6 +106,10 @@ class GenericParameters {
template <typename T, typename = EnableIfValidGenericDataType<T>>
std::vector<std::string> getKeys() const;

/// Get all the available values for a given type
template <typename T, typename = EnableIfValidGenericDataType<T>>
std::vector<std::vector<T>> getValues() const;

/// erase all elements
void clear() {
_intMap.clear();
Expand Down Expand Up @@ -243,5 +248,19 @@ std::vector<std::string> GenericParameters::getKeys() const {
return keys;
}

template <typename T, typename>
std::vector<std::vector<T>> GenericParameters::getValues() const {
std::vector<std::vector<T>> values;
auto& mtx = getMutex<T>();
const auto& map = getMap<T>();
{
// Lock to avoid concurrent changes to the map while we get the stored
// values
std::lock_guard lock{mtx};
values.reserve(map.size());
std::transform(map.begin(), map.end(), std::back_inserter(values), [](const auto& pair) { return pair.second; });
}
return values;
}
} // namespace podio
#endif
2 changes: 0 additions & 2 deletions include/podio/RNTupleReader.h
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
#ifndef PODIO_RNTUPLEREADER_H
#define PODIO_RNTUPLEREADER_H

#include "podio/CollectionBranches.h"
#include "podio/ICollectionProvider.h"
#include "podio/ROOTFrameData.h"
#include "podio/SchemaEvolution.h"
#include "podio/podioVersion.h"
Expand Down
56 changes: 28 additions & 28 deletions include/podio/RNTupleWriter.h
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
#ifndef PODIO_RNTUPLEWRITER_H
#define PODIO_RNTUPLEWRITER_H

#include "podio/CollectionBase.h"
#include "podio/Frame.h"
#include "podio/GenericParameters.h"
#include "podio/SchemaEvolution.h"
#include "podio/utilities/DatamodelRegistryIOHelpers.h"
#include "podio/utilities/RootHelpers.h"

#include "TFile.h"
#include <ROOT/RNTuple.hxx>
Expand Down Expand Up @@ -102,42 +102,42 @@ class RNTupleWriter {
checkConsistency(const std::vector<std::string>& collsToWrite, const std::string& category) const;

private:
template <typename T>
void fillParams(GenericParameters& params, ROOT::Experimental::REntry* entry);
std::unique_ptr<ROOT::Experimental::RNTupleModel>
createModels(const std::vector<root_utils::StoreCollection>& collections);

/// Helper struct to group all the necessary information for one category.
struct CategoryInfo {
std::unique_ptr<ROOT::Experimental::RNTupleWriter> writer{nullptr}; ///< The RNTupleWriter for this category

// The following are assumed to run in parallel!
std::vector<uint32_t> ids{}; ///< The ids of all collections
std::vector<std::string> names{}; ///< The names of all collections
std::vector<std::string> types{}; ///< The types of all collections
std::vector<short> subsetCollections{}; ///< The flags identifying the subcollections
std::vector<SchemaVersionT> schemaVersions{}; ///< The schema versions of all collections

// Storage for the keys & values of all the parameters of this category
// (resp. at least the current entry)
root_utils::ParamStorage<int> intParams{};
root_utils::ParamStorage<float> floatParams{};
root_utils::ParamStorage<double> doubleParams{};
root_utils::ParamStorage<std::string> stringParams{};
};
CategoryInfo& getCategoryInfo(const std::string& category);

using StoreCollection = std::pair<const std::string&, podio::CollectionBase*>;
std::unique_ptr<ROOT::Experimental::RNTupleModel> createModels(const std::vector<StoreCollection>& collections);
template <typename T>
void fillParams(const GenericParameters& params, CategoryInfo& catInfo, ROOT::Experimental::REntry* entry);

std::unique_ptr<ROOT::Experimental::RNTupleModel> m_metadata{};
std::unique_ptr<ROOT::Experimental::RNTupleWriter> m_metadataWriter{};
template <typename T>
root_utils::ParamStorage<T>& getParamStorage(CategoryInfo& catInfo);

std::unique_ptr<TFile> m_file{};

DatamodelDefinitionCollector m_datamodelCollector{};

struct CollectionInfo {
std::vector<uint32_t> id{};
std::vector<std::string> name{};
std::vector<std::string> type{};
std::vector<short> isSubsetCollection{};
std::vector<SchemaVersionT> schemaVersion{};
std::unique_ptr<ROOT::Experimental::RNTupleWriter> writer{nullptr};
};
CollectionInfo& getCategoryInfo(const std::string& category);

std::unordered_map<std::string, CollectionInfo> m_categories{};
std::unordered_map<std::string, CategoryInfo> m_categories{};

bool m_finished{false};

std::vector<std::string> m_intkeys{}, m_floatkeys{}, m_doublekeys{}, m_stringkeys{};

std::vector<std::vector<int>> m_intvalues{};
std::vector<std::vector<float>> m_floatvalues{};
std::vector<std::vector<double>> m_doublevalues{};
std::vector<std::vector<std::string>> m_stringvalues{};

template <typename T>
std::pair<std::vector<std::string>&, std::vector<std::vector<T>>&> getKeyValueVectors();
};

} // namespace podio
Expand Down
39 changes: 5 additions & 34 deletions include/podio/ROOTFrameData.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,37 +25,15 @@ class ROOTFrameData {
ROOTFrameData(const ROOTFrameData&) = delete;
ROOTFrameData& operator=(const ROOTFrameData&) = delete;

ROOTFrameData(BufferMap&& buffers, CollIDPtr&& idTable, podio::GenericParameters&& params) :
m_buffers(std::move(buffers)), m_idTable(std::move(idTable)), m_parameters(std::move(params)) {
}
ROOTFrameData(BufferMap&& buffers, CollIDPtr&& idTable, podio::GenericParameters&& params);

std::optional<podio::CollectionReadBuffers> getCollectionBuffers(const std::string& name) {
const auto bufferHandle = m_buffers.extract(name);
if (bufferHandle.empty()) {
return std::nullopt;
}
std::optional<podio::CollectionReadBuffers> getCollectionBuffers(const std::string& name);

return {bufferHandle.mapped()};
}
podio::CollectionIDTable getIDTable() const;

podio::CollectionIDTable getIDTable() const {
// Construct a copy of the internal table
return {m_idTable->ids(), m_idTable->names()};
}
std::unique_ptr<podio::GenericParameters> getParameters();

std::unique_ptr<podio::GenericParameters> getParameters() {
return std::make_unique<podio::GenericParameters>(std::move(m_parameters));
}

std::vector<std::string> getAvailableCollections() const {
std::vector<std::string> collections;
collections.reserve(m_buffers.size());
for (const auto& [name, _] : m_buffers) {
collections.push_back(name);
}

return collections;
}
std::vector<std::string> getAvailableCollections() const;

private:
// TODO: switch to something more elegant once the basic functionality and
Expand All @@ -66,13 +44,6 @@ class ROOTFrameData {
podio::GenericParameters m_parameters{};
};

// Interim workaround for https://github.com/AIDASoft/podio#500
inline ROOTFrameData::~ROOTFrameData() {
for (auto& [_, buffer] : m_buffers) {
buffer.deleteBuffers(buffer);
}
}

} // namespace podio

#endif // PODIO_ROOTFRAMEDATA_H
5 changes: 2 additions & 3 deletions include/podio/ROOTLegacyReader.h
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
#ifndef PODIO_ROOTLEGACYREADER_H
#define PODIO_ROOTLEGACYREADER_H

#include "podio/CollectionBranches.h"
#include "podio/ROOTFrameData.h"
#include "podio/podioVersion.h"
#include "podio/utilities/RootHelpers.h"

#include "TChain.h"

#include <iostream>
#include <memory>
#include <string>
#include <tuple>
Expand Down Expand Up @@ -114,7 +113,7 @@ class ROOTLegacyReader {
private:
std::pair<TTree*, unsigned> getLocalTreeAndEntry(const std::string& treename);

void createCollectionBranches(const std::vector<std::tuple<uint32_t, std::string, bool, unsigned int>>& collInfo);
void createCollectionBranches(const std::vector<root_utils::CollectionWriteInfoT>& collInfo);

podio::GenericParameters readEventMetaData();

Expand Down
3 changes: 1 addition & 2 deletions include/podio/ROOTReader.h
View file
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
#ifndef PODIO_ROOTREADER_H
#define PODIO_ROOTREADER_H

#include "podio/CollectionBranches.h"
#include "podio/ROOTFrameData.h"
#include "podio/podioVersion.h"
#include "podio/utilities/DatamodelRegistryIOHelpers.h"
#include "podio/utilities/RootHelpers.h"

#include "TChain.h"

#include <iostream>
#include <memory>
#include <string>
#include <string_view>
Expand Down
22 changes: 8 additions & 14 deletions include/podio/ROOTWriter.h
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#ifndef PODIO_ROOTWRITER_H
#define PODIO_ROOTWRITER_H

#include "podio/CollectionBranches.h"
#include "podio/CollectionIDTable.h"
#include "podio/utilities/DatamodelRegistryIOHelpers.h"
#include "podio/utilities/RootHelpers.h"

#include "TFile.h"

Expand Down Expand Up @@ -100,31 +100,25 @@ class ROOTWriter {
checkConsistency(const std::vector<std::string>& collsToWrite, const std::string& category) const;

private:
using StoreCollection = std::pair<const std::string&, podio::CollectionBase*>;

// collectionID, collectionType, subsetCollection
// @note same as in rootUtils.h private header!
using CollectionInfoT = std::tuple<uint32_t, std::string, bool, unsigned int>;

/// Helper struct to group together all necessary state to write / process a
/// given category. Created during the first writing of a category
struct CategoryInfo {
TTree* tree{nullptr}; ///< The TTree to which this category is written
std::vector<root_utils::CollectionBranches> branches{}; ///< The branches for this category
std::vector<CollectionInfoT> collInfo{}; ///< Collection info for this category
podio::CollectionIDTable idTable{}; ///< The collection id table for this category
std::vector<std::string> collsToWrite{}; ///< The collections to write for this category
TTree* tree{nullptr}; ///< The TTree to which this category is written
std::vector<root_utils::CollectionBranches> branches{}; ///< The branches for this category
std::vector<root_utils::CollectionWriteInfoT> collInfo{}; ///< Collection info for this category
podio::CollectionIDTable idTable{}; ///< The collection id table for this category
std::vector<std::string> collsToWrite{}; ///< The collections to write for this category
};

/// Initialize the branches for this category
void initBranches(CategoryInfo& catInfo, const std::vector<StoreCollection>& collections,
void initBranches(CategoryInfo& catInfo, const std::vector<root_utils::StoreCollection>& collections,
/*const*/ podio::GenericParameters& parameters);

/// Get the (potentially uninitialized category information for this category)
CategoryInfo& getCategoryInfo(const std::string& category);

static void resetBranches(std::vector<root_utils::CollectionBranches>& branches,
const std::vector<ROOTWriter::StoreCollection>& collections,
const std::vector<root_utils::StoreCollection>& collections,
/*const*/ podio::GenericParameters* parameters);

std::unique_ptr<TFile> m_file{nullptr}; ///< The storage file
Expand Down
Loading

0 comments on commit 462e88c

Please sign in to comment.