From 10c2a07043be073cba625b1bbb78a669c0ad1442 Mon Sep 17 00:00:00 2001 From: Anthony Scopatz Date: Fri, 5 Jul 2013 15:16:52 -0500 Subject: [PATCH] draft CEP17 --- source/cep/cep0.rst | 7 + source/cep/cep17.rst | 605 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 612 insertions(+) create mode 100644 source/cep/cep17.rst diff --git a/source/cep/cep0.rst b/source/cep/cep0.rst index 5c2d90ebf..608da312c 100644 --- a/source/cep/cep0.rst +++ b/source/cep/cep0.rst @@ -26,6 +26,13 @@ Index P - 1 - CEP Purpose and Guidelines - Scopatz +*Standard CEPs* + +.. toctree:: + :maxdepth: 1 + + SD - 17 - Resource Tracking and Interfaces Re-Re-Redo - Carlsen + Key --- - S - Standards Track CEP diff --git a/source/cep/cep17.rst b/source/cep/cep17.rst new file mode 100644 index 000000000..5653b5251 --- /dev/null +++ b/source/cep/cep17.rst @@ -0,0 +1,605 @@ +CEP 17 - Resource Tracking and Interfaces Re-Re-Redo +*********************************************************** + +:CEP: 17 +:Title: Resource Tracking and Interfaces Re-Re-Redo +:Last-Modified: 2013-07-05 +:Author: Robert Carlsen +:Status: Draft +:Type: Standards Track +:Created: Robert Carlsen + +Abstract +=========== + +This proposal serves to address two related issues: + +1. **Resource tracking:** + + Several types of analysis are made difficult to impossible by the + incomplete resource history currently recorded by Cyclus simulations. Such + analyses include agent time-dependent isotopic inventories and + proliferation-resistance material handling. It is proposed that all + resource state and state transitions will be tracked (as opposed to just + recording state at inter-agent transactions). In addition to required + internal additions/changes to resource classes, explicit `transmute` + functionality will be added to the Material class to accommodate + mass-conserving state changes that occur in agents such as Reactors. + +2. **Resource interface complexity:** + + Creating, querying, and manipulating resources is too complex and is spread + across too many classes. The resource class (and friends) will have their + interfaces minimized and all extra functionality will be provided by one or + more wrapper classes. + +Motivation and Rationale +========================== + +This proposal serves to address two related issues: resource tracking and +resource interface complexity. Because changes to resource tracking will be +intrusive to the current resource interfaces, performing these changes +together is desirable. + +**Resouce Tracking:** + +Currently Cyclus only tracks the state of resources when they are transacted. +This limited tracking is problematic for certain kinds of output +analysis. Examples: + +1. The meaning of "inventory" for agents that transmute materials is + ill-defined. To calculate the inventory of a reactor faciltiy, an + analysis of present data would show fresh fuel coming in, and spent fuel + coming out. The aggregation of transactions to a specific point in time + would not be the reactor's current inventory. Rather it would be the + reactor's inventory plus the difference between fresh and spent fuel + processed until that point in time. The same problem exists with decay. + +2. We cannot determine if a resource was ever handled inappropriately (e.g. + non-proliferation analysis) internally within an agent. + +3. We are unable to determine "source" agents that create material and what + they created. + +4. There is no mass-conservation-safe way to transmute a material (i.e. + change its nuclide composition) outside of destroying and simultaneously + creating new material objects. + +5. When resources are split and combined, there is no way to recall this + heritage. Tracing paths of certain material objects is not possible. + Analyzing the contribution of agent X to the flow of element Y through + agent Z is not possible. + +The above deficiencies are used as drivers for designing/implementing a more +thorough resource tracking infrastructure. + +Additionally, some materials will exist in the simulation that are not +"part" of the simulation and should not be recorded in the ouptut. Some of +these materials will be objects representing what agents are +offering/requesting. A new implementation must accomodate this +requirement. + +**Resource Interface Complexity:** + +The interface(s) for creating, manipulating, and using resources is too +confusing and complex. There are methods for accessing material state +variables spread across three classes (Material, IsoVector, CompMap). The +current trio will be reduced to a duo (Material and Composition classes). +All query convenience related methods will be moved to one (or more) +wrapper classes for interrogating and manipulating material compositions. +*Functionality provided by these wrapper classes can grow organically as +needs arise without affecting the core resource classes.* + +Specification +=============== + +A reference implementation exists for both the cyclus core and cycamore. +These reside at https://github.com/rwcarlsen/cyclus ("res" branch) and +https://github.com/rwcarlsen/cycamore ("res" branch) respectively. Further +implementation details can be found there. + +Data Tracking ++++++++++++++++++++++++ + +In addition to tracking the transfer of resources between agents (and +corresponding state), we track: + +* Transmutation of a material resource within an agent (Resource, + ResourceHeritage, Compositions tables). This helps address problem + #1 and #4 from Motivation section. + +* Decay of a material resource (Resource, ResourceHeritage, Compositions + tables): This is a special, common case of transmutation. + +* All splitting/combining of materials. Tracked by recording the parent(s) + of each Resource object. Resources with no parent were newly created. + Helps address problem #2, #3 and #5. + +This also simplifies rules for acceptable resource handling, namely, it is +never okay for a resource to be destructed or thrown-away unless it is +being stored permanently. The new changes should make it more obvious to +agent developers how enforce correct, measurable mass-conservation. + +Output Schema ++++++++++++++++++++++++ + +All recorded data will stay the same except for the tables listed below: + +* [TableName] ([new/modified/removed]): [field1], [field2], ... + +- Resource (modified): ID, type, quantity, units, StateID, Parent1, Parent2 +- Compositions (new): ID, Isotope, Quantity +- TransactedResources (modified): TransactionID, Position, ResourceID +- GenericResources (removed) +- IsotopicStates (removed) + +*Note that we no longer need a special table for Generic resources - it is +absorbed into the new "Resources" table.* + +Resources/Material API ++++++++++++++++++++++++ + +The Material and Composition classes will be designed to provide only the +minimal interface to support basic manipulation and tracking required by the +cyclus core. All more complex operations will be implemented in helper +classes (like MatQuery). A summary of each of these classes' new role and +its public+protected+private interfaces are listed below. + +Resource class +~~~~~~~~~~~~~~~ + +Resource class provides an abstract interface allowing different types of +resources to be transacted in a simulation. It handles some basic state +tracking and output recording assisted by method invocations from its +subclasses. + +.. code-block:: c++ + + typedef std::string ResourceType; + + class Resource { + public: + typedef boost::shared_ptr Ptr; + + virtual ~Resource(); + + /// Unique for each material object. Changes whenever *any* state changing + /// operation is made. + const int ID(); + + /// Returns the units this resource is based in. + virtual std::string units() = 0; + + /// returns the quantity of this resource with dimensions as specified by units(). + virtual double quantity() = 0; + + /// splits the resource and returns the extracted portion as a new resource + /// object. Allows for things like ResourceBuff and market matching to + /// split offers/requests of arbitrary resource implementation type. + virtual Ptr extractRes(double quantity) = 0; + + virtual ResourceType type() = 0; + + /// returns an untracked (not part of the simulation) copy of the resource. + virtual Ptr clone() = 0; + // the clone method implementations should set tracked_ = false. + + /// friends allow setting of tracked_ param when cloning in subclasses / + /// without making it public. And also allow calling of changeState in create + /// factory functions (wouldn't work even if protected because not changing + /// on context "this". + friend class GenericResource; + friend class Material; + + protected: + Resource(); + + /// records the resource's state that is not accessible via the Resource / + /// class interface (e.g. don't record units, quantity, etc) in its own + /// table. + virtual void recordState() = 0; + + /// returns an id representing the specific resource implementation's internal state. + virtual int stateId() = 0; + + + private: + /// called by subclasses whenever any state changing operation has been + /// performed. Updates the ID and recordes the resources state in the output + /// database. + void changeState(int parent1, int parent2 = 0); + + void recordRes(); + + static int nextId_; + int id_; + bool tracked_; + + int parent1_; + int parent2_; + }; + +Material class +~~~~~~~~~~~~~~~ + +The material class is primarily responsible for enabling basic material +manipulation while helping enforce mass conservation. It also provides the +ability to easily decay a material up to the current simulation time; it +does not perform any decay related logic itself. + +.. code-block:: c++ + + class Material: public Resource { + public: + typedef boost::shared_ptr Ptr; + static ResourceType Type; + + static Ptr create(double quantity, Composition::Ptr c); + static Ptr createUntracked(double quantity, Composition::Ptr c); + + virtual ~Material(); + + /// returns "kg" + virtual std::string units(); + + /// returns the mass of this material in kg. + virtual double quantity(); + + virtual ResourceType type(); + + virtual int stateId(); + + virtual Resource::Ptr clone(); + + virtual Resource::Ptr extractRes(double qty); + + Ptr extractQty(double qty); + + Ptr extractComp(double qty, Composition::Ptr c); + + void absorb(Ptr mat); + + void transmute(Composition::Ptr c); + + Composition::Ptr comp(); + + void decay(int curr_time); + + static void decayAll(int curr_time); + + protected: + virtual void recordState(); + + Material(double quantity, Composition::Ptr c); + + private: + Composition::Ptr mix(double other_qty, Composition::Ptr other); + + double qty_; + Composition::Ptr comp_; + int prev_decay_time_; + static std::map all_mats_; + }; + +GenericResource class +~~~~~~~~~~~~~~~~~~~~~~ + +Implements the Resource class interface. No "quality" property is +provided. It seemed redundant with "units". For a resource of quality +"bananas", just use units like "kg bananas" (for mass based) or "# bananas" +(for count based). + +.. code-block:: c++ + + class GenericResource : public Resource { public: + typedef boost::shared_ptr Ptr; + static ResourceType Type; + + static Ptr create(double quantity, std::string units); + static Ptr createUntracked(double quantity, std::string units); + + /// Returns a reference to a newly allocated copy of this resource + virtual Resource::Ptr clone(); + + /// Returns the total quantity of this resource in its base unit + virtual double quantity() {return quantity_;}; + + /// Returns the total quantity of this resource in its base unit + virtual std::string units() {return units_;}; + + /// Returns the concrete type of this resource + virtual ResourceType type() {return Type;}; + + /// not needed/no meaning for generic resources + virtual int stateId() {return 0;}; + + /** + Absorbs the contents of the given 'other' resource into this + resource + @throws CycGenResourceIncompatible 'other' resource is of a + */ + virtual void absorb(GenericResource::Ptr other); + + /** + Extracts the specified mass from this resource and returns it as a + new generic resource object with the same quality/type. + + @throws CycGenResourceOverExtract + */ + GenericResource::Ptr extract(double quantity); + + virtual Resource::Ptr extractRes(double quantity); + + protected: + + virtual void recordState() { }; + + private: + + /** + @param quantity is a double indicating the quantity + @param units is a string indicating the resource unit + */ + GenericResource(double quantity, std::string units); + + /** + The units of the resource + */ + std::string units_; + + /** + The quantity of the resource + */ + double quantity_; + }; + +Composition class +~~~~~~~~~~~~~~~~~~~~~~ + +An immutable object responsible for tracking decay lineages (to prevent +duplicate calculations and output recording) and able to record its +composition data to output when told. Each composition will keep a pointer +to references to every other composition that is a result of decaying this +or a previously decayed-from composition. + +Note that previously, composition creation/modification involved a notion +of equivalence via threshold comparison to facilitate reduced +memory/storage burdens. This proposal discards this idea in favor of +defining equivalence trivially as "the same object in memory" or pointer +equality. Some discussion regarding this can be found in comments here: +https://github.com/cyclus/cyclus/issues/484. Of particular concern w.r.t. +the previous equivalence notion is this:: + + Also - another potential issue I thought of: Repeatedly calling multiple + consecutive small differences negligible could result in compositions + staying the same that would have otherwise been appreciably different if + each small change were allowed to propogate as a new composition. + +.. code-block:: c++ + + class Composition { + public: + typedef boost::shared_ptr Ptr; + typedef std::map Vect; + + static Ptr createFromAtom(Vect v); + static Ptr createFromMass(Vect v); + + int ID(); + + Ptr decay(int delta); + + const Vect& atomVect(); + const Vect& massVect(); + + /// record in output database (if not done previously). + void record(); + + protected: + Composition(); + + typedef std::map Chain; + typedef boost::shared_ptr ChainPtr; + ChainPtr decay_line_; + + private: + // This constructor allows the creation of decayed versions of + // compositions while avoiding extra memory allocations. + Composition(int prev_decay, ChainPtr decay_line); + + Ptr newDecay(int delta); + + // normalizes the sum of all quantities in the composition's vector to one. + void normalize(Vect& v); + + static int nextId_; + + int id_; + bool recorded_; + Vect atomv_; + Vect massv_; + int prev_decay_; + }; + +MatQuery class +~~~~~~~~~~~~~~~~~~~~~~ + +(This interface will probably need extension) + +Will be designed to allow user-developers to *easily* retrieve any kind of +information about a material they could ever reasonably need. + +.. code-block:: c++ + + class MatQuery { + public: + MatQuery(Material::Ptr m) : m_(m) { } + + double mass(Iso iso) { + return massFrac(iso) * qty(); + } + + double moles(Iso iso) { + return mass(iso) / (MT->gramsPerMol(iso) * units::g); + } + + double massFrac(Iso iso) { + Composition::Vect v = m_->comp()->massVect(); + return v[iso]; + }; + + double atomFrac(Iso iso) { + Composition::Vect v = m_->comp()->atomVect(); + return v[iso]; + }; + + double qty() { + return m_->quantity(); + }; + + private: + + Material::Ptr m_; + }; + +Other Changes +++++++++++++++ + +The RecipeLibrary's role of composition decay management has been shifted +into the Composition class. It now is only responsible for loading +recipes from xml input and serving them up simulation wide. Agents are +also allowed to register their own compositions manually. + +*The decay lineage tracking functionality introduced by Matt Gidden has been +effectively preserved.* + +.. code-block:: c++ + + class RecipeLibrary { + public: + /** + Gives all simulation objects global access to the RecipeLibrary by + returning a pointer to it. + Like the Highlander, there can be only one. + + @return a pointer to the RecipeLibrary + */ + static RecipeLibrary* Instance(); + + /** + loads the recipes from the input file + */ + void load_recipes(QueryEngine* qe); + + /** + loads a specific recipe + */ + void load_recipe(QueryEngine* qe); + + /** + records a new recipe in the simulation + - records the recipe in the BookKeeper + + @param recipe the recipe to be recorded, a CompMapPtr + */ + void addRecipe(std::string name, Composition::Ptr c); + + /** + This returns a CompMapPtr to the named recipe in the recipes_ map + + @param name the name of the parent recipe, a key in the recipes_ map + */ + Composition::Ptr getRecipe(std::string name); + + private: + RecipeLibrary(); + + /// A pointer to this RecipeLibrary once it has been initialized. + static RecipeLibrary* instance_; + + RecipeMap recipes_; + }; + + + +Backwards Compatibility +======================== + +Most backwards incompatible changes are unambiguously described by the +reference implementation at https://github.com/rwcarlsen/cycamore ("res" +branch). Existing modules will need to be updated to use the new API's. These +changes are fairly straight forward and include: + +* Material queries will have to be modified to use MatQuery class. + +* CompMap/IsoVector creation will need to change to use new Composition + factory methods. + +* Material creation will need to change to use new Material factory + methods. + +* Agents (esp. reactors) must be modified to transmute rather than + throw-away/create material. + +Other Notes +============ + +Current implementation bugs +---------------------------- + +The current (before this CEP) Cyclus core does not correctly record decayed +compositions in the output database. This has the effect of comparing +simulation output size and performance with that of this CEP's proposed +changes is not exactly "fair". + +Backends and Performance +------------------------- + +Preliminary investigation on my part indicates that this extra tracking +will cause significant slowdown using an Sqlite backend database *when +material decay is frequent*. This slowdown prompted the development of a +faster HDF5 alternative. This alternate backend currently lives at +https://github.com/rwcarlsen/cyclus ("hdf5" branch). + +Basic performance stats were collected by running a full cyclus +inpro_low.xml simulation `time cyclus [path/to/]inpro_low.xml`. For +reference: + +* ~50,000 material objects total +* 1100 months +* 2200 decay calculations (2200 separate compositions recorded) +* ~28,000,000 resource object state changes recorded. + +Cyclus was built with CMake's "RELEASE" mode. Results reported are +approximate and specific to my office computer. + +Without proposed changes (decayed compositions are not recorded - current bug): + +=========================== ========= ======== + Sqlite Hdf5 +=========================== ========= ======== +Decay every 2nd timestep 41 sec. 17 sec. +No decay 41 sec. 17 sec. +=========================== ========= ======== + +With proposed changes: + +=========================== ========= =============== + Sqlite Hdf5 +=========================== ========= =============== +Decay every 2nd timestep 16 min. 2 min. 50 sec. +No decay 55 sec. 21 sec. +=========================== ========= =============== + +Decay Initiation +----------------- + +There has been some debate regarding the best way(s) to handle decaying +material objects in the simulation. Options include: manually by agents, +automatically and periodic, automatically at transaction time, and others. +While this involves the resource+material classes and can have a large +impact on simulation speed and output size, it has no direct impact on nor +directly impacts this proposal. Further discussion on this can be found +here https://github.com/cyclus/cyclus/issues/466 and to lesser degree +https://github.com/cyclus/cyclus/issues/204. +