Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New simulation time series plugin #867

Merged
Show file tree
Hide file tree
Changes from 98 commits
Commits
Show all changes
107 commits
Select commit Hold shift + click to select a range
8714f5e
Added pyarrow dependency
sigurdp May 25, 2021
3019582
Utility class for measuring elapsed time
sigurdp May 25, 2021
803e6d9
WIP, first cut impl after spike and redesign
sigurdp May 25, 2021
1a0bb59
Added proper implementation of create_provider_set_from_aggregated_cs…
sigurdp May 25, 2021
63dcf4b
Added test code
sigurdp May 26, 2021
6fd9e8b
Try out compression for arrow backing
sigurdp May 28, 2021
1b52aba
Separate function for making hash string
sigurdp May 28, 2021
5502cbe
Merge branch 'master' into sigurdp/summary-provider
sigurdp Jul 7, 2021
11832e0
Added resampling utility functions
sigurdp Jul 16, 2021
fe9dc5c
Removed loading of per realization smry data via fmu
sigurdp Jul 16, 2021
065a0a2
Minor style fixes
sigurdp Jul 18, 2021
455a89f
Added sample_single_real_table_at_date()
sigurdp Jul 19, 2021
2c5bbc9
Added experimental lazy sampling data provider; currently faking impo…
sigurdp Jul 19, 2021
3af3283
Introduced sampling frequency in factory method for LAZY sampling pro…
sigurdp Jul 19, 2021
e35b199
Test code for accessing path and real number through FMU ensamble
sigurdp Aug 23, 2021
06225f6
Report time it takes to resample
sigurdp Aug 23, 2021
3ee34f7
Comments
sigurdp Aug 23, 2021
996a555
Merge branch 'master' into sigurdp/summary-provider
sigurdp Aug 23, 2021
c9b1e13
Minor fixes during testing
sigurdp Aug 23, 2021
065cb35
wip
sigurdp Aug 23, 2021
d7cd0a3
Typings for Python 3.6
sigurdp Aug 24, 2021
a4d9676
Optimized impl of sampling of all vectors for specific date
sigurdp Aug 27, 2021
7fde7e3
Added method EnsembleSummaryProvider.vector_metadata()
sigurdp Sep 7, 2021
6446457
Merge branch 'master' into provider-based-simulation-time-series
sigurdp Sep 20, 2021
522a348
Merge branch 'provider-based-simulation-time-series' into sigurdp/sum…
sigurdp Sep 21, 2021
8723027
wip: removed fmu unsmry support, started refactor and consolidation
sigurdp Sep 22, 2021
d68b98d
Start incorporating interface changes to EnsembleSummaryProvider
sigurdp Sep 23, 2021
37b9fab
Added guarding against empty list in vector_names parameter
sigurdp Sep 24, 2021
cccfac6
EnsembleSummaryProvider.dates() now returns intersection of available…
sigurdp Sep 27, 2021
910f768
Added imports for summary provider
sigurdp Sep 29, 2021
127c613
Minor adjustment to logging
sigurdp Sep 29, 2021
7180f14
Removed EnsembleSummaryProviderSet, factory methods now return just a…
sigurdp Sep 29, 2021
f940e22
Added test content
sigurdp Sep 29, 2021
4a2568d
Set version requirement of pyarrow to the latest available version.
sigurdp Oct 1, 2021
4b6e45b
Linting and friends
sigurdp Oct 1, 2021
44e5f3d
Linting
sigurdp Oct 1, 2021
d0783e1
Refactor
sigurdp Oct 1, 2021
70bb0a2
Refactor
sigurdp Oct 1, 2021
e0e0648
Renamed sampling functions and extended tests
sigurdp Oct 1, 2021
11a12bc
Renamed dev code file
sigurdp Oct 4, 2021
f0b9063
Extended dev comparison and added resampling perf test
sigurdp Oct 4, 2021
2625241
Minor modifications during testing
sigurdp Oct 8, 2021
2d633b0
Linting
sigurdp Oct 11, 2021
46b297b
Merge branch 'master' into sigurdp/summary-provider
sigurdp Oct 11, 2021
737d57b
Adjusted to new webviz-subsurface-testdata
sigurdp Oct 11, 2021
b5062c9
Initial implementation of time series plugin
jorgenherje Oct 11, 2021
0f545ec
Make EnsembleSummaryProviderFactory obey app's run mode
sigurdp Oct 11, 2021
04918a1
Introduced graph figure builder and rafactored statistics plotting
jorgenherje Oct 14, 2021
20af5f5
Comments
sigurdp Oct 20, 2021
91166c8
Create ensemble set model
jorgenherje Oct 22, 2021
0a18da3
Implemented ProviderSet object
jorgenherje Oct 25, 2021
7a442f6
Add items() iterator for ProviderSet and update naming
jorgenherje Oct 26, 2021
8a0b053
Split interpretation of field metadata into separate module
sigurdp Oct 28, 2021
2f746f2
Added tests for retrieving vector metadata
sigurdp Oct 28, 2021
9edd72e
Import organization
sigurdp Oct 29, 2021
4bc7332
EnsembleSummaryProvider.vector_metadata() now returns dataclass inste…
sigurdp Oct 29, 2021
aa4e017
Added code for reading reference DF using ecl2df
sigurdp Oct 29, 2021
ebc5972
Added code for interpreting flat vector metadata in field's metadata
sigurdp Oct 29, 2021
d581bbe
Add verification of consistent vector metadata for provider set
jorgenherje Nov 5, 2021
6b40dc4
Initial implementation of cumulative calucaltion
jorgenherje Nov 5, 2021
a7e61c2
Furhter refactoring
jorgenherje Nov 9, 2021
87e4767
Added handling of vector metadata as produced by ecl2df
sigurdp Nov 10, 2021
48d2c90
Merge branch 'master' into sigurdp/summary-provider
sigurdp Nov 10, 2021
5f704d9
Fix to test new version of ecl
sigurdp Nov 11, 2021
7f9fa92
Further refactoring for enhanced interface
jorgenherje Nov 11, 2021
4b2d850
Adaptions during testing of bugfixed version of ecl
sigurdp Nov 11, 2021
0fa72c1
Linting and formatting
sigurdp Nov 12, 2021
931763e
Merge branch 'master' into sigurdp/summary-provider
sigurdp Nov 12, 2021
57ee0c9
Refactored plotting utils into GraphFigureBuilder
jorgenherje Nov 12, 2021
dd278af
Further fix of resampling frequency selection
jorgenherje Nov 15, 2021
d56558b
Documentation and extended unit tests
sigurdp Nov 16, 2021
793de27
Pylint, optimization and removal of unused/experimental resampling code
sigurdp Nov 16, 2021
53be485
Merge branch 'master' into sigurdp/summary-provider
sigurdp Nov 16, 2021
50c8a91
Sorting of imports
sigurdp Nov 16, 2021
339b045
Removed ert from test require
sigurdp Nov 16, 2021
7ed158f
Adjust interval delta and average rate calculation frequency and add …
jorgenherje Nov 16, 2021
152c160
Merge remote-tracking branch 'CeetronSolutions/sigurdp/summary-provid…
jorgenherje Nov 16, 2021
f26a1c5
Adjust usage of vector metadata to new dataclass implementation
jorgenherje Nov 16, 2021
3bcb9e4
Renamed and moved files/objects to pursue boilerplate best practice.
jorgenherje Nov 17, 2021
be22c0e
Fix TypedDict import for python version < 3.8
jorgenherje Nov 17, 2021
82ec134
Add VectorCalculator to plugin
jorgenherje Nov 19, 2021
0577f26
Update settings menu layout according to feedback
jorgenherje Nov 22, 2021
eea95dd
Create graph data changed callback trigger
jorgenherje Nov 22, 2021
f482e77
Add units to title for calculated expressions
jorgenherje Nov 22, 2021
c22ee46
Replace DeltaEnsembleProvider with data accessors. Add data download
jorgenherje Nov 25, 2021
ef12f8b
Rename accessors for clarity and utility for creating accessors in ca…
jorgenherje Nov 26, 2021
1b847f2
Move "factory" methods for ProviderSet and clean-up
jorgenherje Nov 29, 2021
d494fc6
Add subplot per ensemble - subplot "group by"
jorgenherje Dec 3, 2021
fc6b207
Test commit for token check
jorgenherje Dec 3, 2021
3b0f82d
Minor update of figure builders
jorgenherje Dec 6, 2021
0b3b865
Update creating legends for graph figures
jorgenherje Dec 7, 2021
04fbd0a
Fix documentation
jorgenherje Dec 7, 2021
9abcb23
Merge remote-tracking branch 'CeetronSolutions/master' into EQ_698-si…
jorgenherje Dec 7, 2021
9e22623
Fix imports
jorgenherje Dec 7, 2021
b615f88
Fix isort errors
jorgenherje Dec 7, 2021
d464cba
Merge remote-tracking branch 'CeetronSolutions/master' into EQ_698-si…
jorgenherje Dec 7, 2021
4e4b9bd
Add tour steps
jorgenherje Dec 8, 2021
b2acb85
Merge remote-tracking branch 'CeetronSolutions/master' into EQ_698-si…
jorgenherje Dec 15, 2021
366c3f5
Update workflow to use new simulation time series testdata branch
jorgenherje Dec 15, 2021
5edb051
Clean-up based on review.
jorgenherje Dec 15, 2021
42c095e
Rename vectors accessor
jorgenherje Dec 15, 2021
2d9c58a
Fix isort errors
jorgenherje Dec 15, 2021
750792c
Remove verification of consistent vector metadata and add value error…
jorgenherje Dec 16, 2021
1aa3f27
Replace deprecated package imports
jorgenherje Dec 16, 2021
eaf2a5f
Updated changelog
jorgenherje Dec 16, 2021
0a73c0c
Merge remote-tracking branch 'CeetronSolutions/master' into EQ_698-si…
jorgenherje Dec 16, 2021
a212cd5
Revert workflow
anders-kiaer Dec 16, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
"RftPlotter = webviz_subsurface.plugins:RftPlotter",
"RunningTimeAnalysisFMU = webviz_subsurface.plugins:RunningTimeAnalysisFMU",
"SegyViewer = webviz_subsurface.plugins:SegyViewer",
"SimulationTimeSeries = webviz_subsurface.plugins:SimulationTimeSeries",
"SeismicMisfit = webviz_subsurface.plugins:SeismicMisfit",
"StructuralUncertainty = webviz_subsurface.plugins:StructuralUncertainty",
"SubsurfaceMap = webviz_subsurface.plugins:SubsurfaceMap",
Expand Down
5 changes: 4 additions & 1 deletion webviz_subsurface/_utils/fanchart_plotting.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
@dataclass
class FreeLineData:
"""
Dataclass for defining statistics data for freee line trace in fanchart
Dataclass for defining statistics data for free line trace in fanchart

`Attributes:`
* `name` - Name of statistics data
Expand Down Expand Up @@ -134,6 +134,7 @@ def get_fanchart_traces(
hovertext: str = "",
hovertemplate: Optional[str] = None,
hovermode: Optional[str] = None,
legendrank: Optional[int] = None,
) -> List[Dict[str, Any]]:
"""
Utility function for creating statistical fanchart traces
Expand Down Expand Up @@ -184,6 +185,8 @@ def get_default_trace(statistics_name: str, values: np.ndarray) -> Dict[str, Any
"legendgroup": legend_group,
"showlegend": False,
}
if legendrank:
trace["legendrank"] = legendrank
if not show_hoverinfo:
trace["hoverinfo"] = "skip"
return trace
Expand Down
233 changes: 233 additions & 0 deletions webviz_subsurface/_utils/statistics_plotting.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,233 @@
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional

import numpy as np


@dataclass
class LineData:
"""
Definition of line trace data for statistics plot

`Attributes:`
* `data` - 1D np.array of value data
* `name` - Name of line data
"""

data: np.ndarray
name: str


@dataclass
class StatisticsData:
"""
Dataclass defining statistics data utilized in creation of statistical plot traces

`Attributes:`
* `samples` - Common sample point list for each following value list.
* `free_line` - LineData with name and value data for free line trace in statistics plot
(e.g. mean, median, etc.)
* `minimum` - Optional 1D np.array of minimum value data for statistics plot
* `maximum` - Optional 1D np.array of maximum value data for statistics plot
* `low` - Optional low percentile, name and 1D np.array data for statistics plot
* `mid` - Optional middle percentile, name and 1D np.array data for statistics plot
* `high` - Optional high percentile, name and 1D np.array data for statistics plot

"""

# TODO:
# - Rename mid percentile, find better name?
# - Consider to replace all lines with List[LineData], where each free line must be
# named and provided data.
# - Can then be used for individual realization plots as well?
# - One suggestion: Create base class with: samples: list, free_lines: List[LineData]
# and inherit for "StatisticsData". Base class can be utilized for realization plots?

jorgenherje marked this conversation as resolved.
Show resolved Hide resolved
samples: list = field(default_factory=list)
free_line: Optional[LineData] = None
minimum: Optional[np.ndarray] = None
maximum: Optional[np.ndarray] = None
low: Optional[LineData] = None
high: Optional[LineData] = None
mid: Optional[LineData] = None


def validate_statistics_data(data: StatisticsData) -> None:
"""
Validation of statistics data

Ensure equal length of all statistical data lists and x-axis data list

Raise ValueError if lengths are unequal
"""
if len(data.samples) <= 0:
jorgenherje marked this conversation as resolved.
Show resolved Hide resolved
raise ValueError("Empty x-axis data list in StatisticsData")
if data.free_line is not None and len(data.samples) != len(data.free_line.data):
raise ValueError(
"Invalid statistics mean value data length. len(data.samples) != len(free_line.data)"
)
if data.minimum is not None and len(data.samples) != len(data.minimum):
raise ValueError(
"Invalid statistics minimum value data length. len(data.samples) "
"!= len(data.minimum)"
)
if data.maximum is not None and len(data.samples) != len(data.maximum):
raise ValueError(
"Invalid statistics maximum value data length. len(data.samples) != "
"len(data.maximum)"
)
if data.low is not None and len(data.samples) != len(data.low.data):
raise ValueError(
"Invalid statistics low percentile value data length. len(data.samples) "
"!= len(data.low.data)"
)
if data.mid is not None and len(data.samples) != len(data.mid.data):
raise ValueError(
"Invalid statistics middle percentile value data length. len(data.samples) "
"!= len(data.mid.data)"
)
if data.high is not None and len(data.samples) != len(data.high.data):
jorgenherje marked this conversation as resolved.
Show resolved Hide resolved
raise ValueError(
"Invalid statistics high percentile value data length. "
"len(data.samples) != len(data.high.data)"
)


# pylint: disable=too-many-arguments
# pylint: disable=too-many-locals
def create_statistics_traces(
data: StatisticsData,
color: str,
legend_group: str,
legend_name: Optional[str] = None,
line_shape: str = "linear",
xaxis: str = "x",
yaxis: str = "y",
show_legend: bool = True,
show_hoverinfo: bool = True,
hovertext: str = "",
hovertemplate: Optional[str] = None,
hovermode: Optional[str] = None,
legendrank: Optional[int] = None,
) -> List[Dict[str, Any]]:
"""
Utility function for creating statistical plot traces

Takes `data` containing data for each statistical feature as input, and creates a list of traces
for each feature. Plotly plots traces from front to end of the list, thereby the last trace is
plotted on top.

Note that the data is optional, which implies that only wanted statistical features needs to be
provided for trace plot generation.

The function provides a list of traces: [trace0, tract1, ..., traceN]

Note:
If hovertemplate is proved it overrides the hovertext

Returns:
List of statistical line traces, one for each statistical feature in data input.
[trace0, tract1, ..., traceN].
"""

validate_statistics_data(data)

def get_default_trace(statistics_name: str, values: np.ndarray) -> Dict[str, Any]:
trace = {
"name": legend_name if legend_name else legend_group,
"x": data.samples,
"y": values,
"xaxis": xaxis,
"yaxis": yaxis,
"mode": "lines",
"line": {"width": 1, "color": color, "shape": line_shape},
"legendgroup": legend_group,
"showlegend": False,
}
if legendrank:
trace["legendrank"] = legendrank
if not show_hoverinfo:
trace["hoverinfo"] = "skip"
return trace
if hovertemplate is not None:
trace["hovertemplate"] = hovertemplate + statistics_name
else:
trace["hovertext"] = statistics_name + " " + hovertext
if hovermode is not None:
trace["hovermode"] = hovermode
return trace

traces: List[Dict[str, Any]] = []

# Minimum
if data.minimum is not None:
minimum_trace = get_default_trace(
statistics_name="Minimum",
values=data.minimum,
)
minimum_trace["line"] = {
"color": color,
"shape": line_shape,
"dash": "longdash",
"width": 1.5,
}
traces.append(minimum_trace)

# Low percentile
if data.low is not None:
low_trace = get_default_trace(
statistics_name=data.low.name, values=data.low.data
)
low_trace["line"] = {"color": color, "shape": line_shape, "dash": "dashdot"}
traces.append(low_trace)

# Mid percentile
if data.mid is not None:
mid_trace = get_default_trace(
statistics_name=data.mid.name, values=data.mid.data
)
mid_trace["line"] = {
"color": color,
"shape": line_shape,
"dash": "dot",
"width": 3,
}
traces.append(mid_trace)

# High percentile
if data.high is not None:
high_trace = get_default_trace(
statistics_name=data.high.name, values=data.high.data
)
high_trace["line"] = {"color": color, "shape": line_shape, "dash": "dashdot"}
traces.append(high_trace)

# Maximum
if data.maximum is not None:
maximum_trace = get_default_trace(
statistics_name="Maximum",
values=data.maximum,
)
maximum_trace["line"] = {
"color": color,
"shape": line_shape,
"dash": "longdash",
"width": 1.5,
}
traces.append(maximum_trace)

# Free line
if data.free_line is not None:
line_trace = get_default_trace(
statistics_name=data.free_line.name,
values=data.free_line.data,
)
# Set solid line
line_trace["line"] = {"color": color, "shape": line_shape}
traces.append(line_trace)

# Set legend for last trace in list
if len(traces) > 0:
traces[-1]["showlegend"] = show_legend

return traces
89 changes: 87 additions & 2 deletions webviz_subsurface/_utils/vector_calculator.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import sys
from pathlib import Path
from typing import Dict, List, Optional, Tuple, Union
from typing import Dict, List, Optional, Sequence, Tuple, Union
from uuid import uuid4

import numpy as np
Expand All @@ -14,7 +14,12 @@
VectorCalculator,
)

from .vector_selector import is_vector_name_in_vector_selector_data
from webviz_subsurface._providers import EnsembleSummaryProvider, Frequency

from .vector_selector import (
add_vector_to_vector_selector_data,
is_vector_name_in_vector_selector_data,
)

if sys.version_info >= (3, 8):
from typing import TypedDict
Expand Down Expand Up @@ -276,6 +281,46 @@ def get_calculated_vector_df(
return df[columns + [name]]


def create_calculated_vector_df(
expression: ExpressionInfo,
provider: EnsembleSummaryProvider,
realizations: Optional[Sequence[int]],
resampling_frequency: Optional[Frequency],
) -> pd.DataFrame:
"""Create dataframe with calculated vector from expression

If expression is not successfully evaluated, empty dataframe is returned

`Return:`
* Dataframe with calculated vector data made form expression - columns:\n
["DATE","REAL", calculated_vector]
* Return empty dataframe if expression evaluation returns None
"""
name: str = expression["name"]
expr: str = expression["expression"]

variable_vector_dict: Dict[str, str] = VectorCalculator.variable_vector_dict(
expression["variableVectorMap"]
)
vector_names = list(variable_vector_dict.values())

# Retrieve data for vectors in expression
vectors_df = provider.get_vectors_df(
vector_names, resampling_frequency, realizations
)

values: Dict[str, np.ndarray] = {}
for variable, vector in variable_vector_dict.items():
values[variable] = vectors_df[vector].values

evaluated_expression = VectorCalculator.evaluate_expression(expr, values)
if evaluated_expression is not None:
vectors_df[name] = evaluated_expression
return vectors_df[["DATE", "REAL", name]]

return pd.DataFrame()


@CACHE.memoize(timeout=CACHE.TIMEOUT)
def get_calculated_units(
expressions: List[ExpressionInfo],
Expand Down Expand Up @@ -308,3 +353,43 @@ def get_calculated_units(
except ValueError:
continue
return calculated_units


def add_calculated_vector_to_vector_selector_data(
vector_selector_data: list,
vector_name: str,
description: Optional[str] = None,
) -> None:
"""Add calculated vector name and descritpion to vector selector data

Description is optional, and will be added at last node
"""
description_str = description if description is not None else ""
add_vector_to_vector_selector_data(
vector_selector_data=vector_selector_data,
vector=vector_name,
description=description_str,
description_at_last_node=True,
)


def add_expressions_to_vector_selector_data(
vector_selector_data: list, expressions: List[ExpressionInfo]
) -> None:
"""Add expressions to vector selector data

Adds calculated vector name into node structure. Adds expression
description if existing.
"""
for expression in expressions:
if not expression["isValid"]:
continue

name = expression["name"]
description = None
if "description" in expression.keys():
description = expression["description"]

add_calculated_vector_to_vector_selector_data(
vector_selector_data, name, description
)
1 change: 1 addition & 0 deletions webviz_subsurface/plugins/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
from ._running_time_analysis_fmu import RunningTimeAnalysisFMU
from ._segy_viewer import SegyViewer
from ._seismic_misfit import SeismicMisfit
from ._simulation_time_series import SimulationTimeSeries
from ._structural_uncertainty import StructuralUncertainty
from ._subsurface_map import SubsurfaceMap
from ._surface_viewer_fmu import SurfaceViewerFMU
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from ._plugin import SimulationTimeSeries
Loading