You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
During evaluation, models are assessed for three categories of data: external validation, internal validation and development. At the level of an ensemble of models, each model may have different development and internal validation data. This is currently handled correctly. However, the system is obscure. It depends on data not being loaded at prediction time, and then making decisions based on data_ids and run_ids. That is, this behaviour is baked into the deepest code layers, but is directed from high-level code in an indirect manner.
To make this behaviour more transparent and testable, we need to make this behaviour steerable from the code that generates and configures the evaluation task (enabled in v2.0.0 by switching to a task-based system instead of a purely functional system).
The text was updated successfully, but these errors were encountered:
Added delayedDataObject class to start distinguishing between dataObject that can be processed directly, and delayedDataObject which is intended to populate the dataObject using its associated attributes and the backend. This will allow for separating methods and attributes related to loading data that are currently associated with dataObject.
Rework methods for dataObject that deal with loading data and move these to delayedDataObject.
Use delayedDataObject to contextualise data processing within evaluation steps.
During evaluation, models are assessed for three categories of data: external validation, internal validation and development. At the level of an ensemble of models, each model may have different development and internal validation data. This is currently handled correctly. However, the system is obscure. It depends on data not being loaded at prediction time, and then making decisions based on
data_id
s andrun_id
s. That is, this behaviour is baked into the deepest code layers, but is directed from high-level code in an indirect manner.To make this behaviour more transparent and testable, we need to make this behaviour steerable from the code that generates and configures the evaluation task (enabled in v2.0.0 by switching to a task-based system instead of a purely functional system).
The text was updated successfully, but these errors were encountered: