Merge branch 'main' into change-name-and-tagname-for-grids

docs/index.rst

@@ -10,7 +10,7 @@ The package facilitates upload of results from reservoir simulators such as **ec
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents:
+   :caption: Contents
 
    self
    sim2sumo

docs/sim2sumo.rst

@@ -145,42 +145,49 @@ Custom configuration
 =====================
 
 The sim2sumo section in the config file gives you full flexibility for extracting anything that ``res2df`` can extract.
-You can also change where you extract results from, and even use all the extra custumization options that ``res2df`` has available.
+You can also change where you extract results from, and use some of the extra customization options that ``res2df`` has available.
 The three relevant sections are:
 
-*datafile*:
+datafile
 --------------------
-This section is for configuring where you extract results from, meaning where to look for simulation results. This section can be configured in several ways:
+This section is for configuring where you extract results from, meaning where to look for simulation results.
+This section should be a list where each item is either a file path, file stub or a folder path.
 
-1. As a path to a file, or file stub (without an extension):
+1. File paths, or file stubs (without an extension):
 
-   .. code-block:: yaml
+   .. code-block::
 
-      datafile ../../eclipse/model/DROGON
+      datafile:
+         - ../../eclipse/model/DROGON
+         - ../../eclipse/model/DROGON-0.DATA
 
 
-2. As a path to a folder:
+2. Folder paths:
 
    .. code-block::
 
-      datafile: ../../eclipse/model/
+      datafile:
+         - ../../eclipse/model/
 
 
-3. As a list:
+You can also specify what datatypes should be extracted for each file, by adding a list of datatypes to each file path:
 
    .. code-block::
 
       datafile:
-        - ../../eclipse/model
-        - ../../ix/model
-        ..
+         - ../../eclipse/model/DROGON:
+            - summary
+            - wcon
+            - faults
+         - ../../eclipse/model/DROGON-0.DATA:
+            - summary
+            - wcon
+            - faults
 
 
-datatypes:
+datatypes
 ----------------
-This section is for configuration of what data to extract. The section can be configured in several ways.
-
-1. As list:
+This section is for configuration of what data to extract. It should be specified as a list
 
    .. code-block::
 
@@ -190,22 +197,48 @@ This section is for configuration of what data to extract. The section can be co
         - faults
         - ..
 
-2. as string:
+To include all datatypes use a list with a single item "all":
+
+   .. code-block::
+
+      datatypes:
+         - all
+
+For datatypes available see documentation for ``res2df``
+
+.. options:
+.. -------------
+..    | This section is for adding extra optional configuration for extracting the different datatypes.
+..    | This section needs to be in a list format.
+
+grid3d
+----------------
+When ``grid3d`` is set to ``true``, sim2sumo will try to upload 3D grid data and properties for all datafiles specified in the ``datafile`` section.
+For the init file the following properties will be *ignored*: ENDNUM, DX, DY, DZ, TOPS.
+For the restart the following properties will be *exported*: SWAT, SGAS, SOIL, PRESSURE, SFIPOIL, SFIPGAS.
 
-   Here there are two options, you can use both the name of one single datatype
-   or the 'all' argument for all datatypes:
 
    .. code-block::
-      :caption: extracting all available datatypes from simulation run
 
-      datatypes: all
+      grid3d: true
+
 
-   For datatypes available see documentation for ``res2df``
+Overriding default datatypes
+----------------
+``datatypes`` applies to all datafiles specified in the ``datafile`` section.
+It is possible to override this configuration for individual files.
+The example shows how to only extract summary data from the first file, and all default ``datatypes`` from the second file.
 
-options:
--------------
-   | This section is for adding extra optional configuration for extracting the different datatypes.
-   | This section needs to be in a list format.
+   .. code-block::
+
+      datafile:
+         - ../../eclipse/model/DROGON-0.DATA:
+            - summary
+         - ../../eclipse/model/DROGON-1.DATA
+      datatypes:
+         - summary
+         - wcon
+         - faults
 
 
 Using sim2sumo in scripts
@@ -248,14 +281,6 @@ Extracting the default datatypes with sim2sumo
 
 See also :ref:`preconditions`.
 
-Extracting rft data from specified datafile with sim2sumo
-----------------------------------------------------------------
-
-.. code-block::
-   :caption: Extracting rft
-
-   sum2sumo execute --config_path fmuconfig/output/global_variables.yml --datatype rft --datafile eclipse/model/DROGON-0.DATA
-
 
 Getting help on sim2sumo from the command line
 =================================================
@@ -280,5 +305,24 @@ Accessing help from ``res2df`` via sim2sumo
 
 
 
+VFP export
+***************************************
+**Note** that Sim2sumo exports VFP tables as separate files. (Similar to using ``--arrow`` with ``res2df``)
+
+VFP data can be exported either for all datafiles by adding "vfp" to the datatypes list:
+
+   .. code-block::
+
+      datatypes:
+        - ..
+        - vfp
+
+Or for individual files by adding "vfp" to the list of datatypes for that file:
+
+.. code-block::
 
+      datafile:
+         - ../../eclipse/model/DROGON-0.DATA:
+            - ..
+            - vfp
 

src/fmu/sumo/sim2sumo/_special_treatments.py

@@ -21,8 +21,6 @@ def convert_to_arrow(frame):
     Returns:
         pa.Table: the converted dataframe
     """
-    logger = logging.getLogger(__file__ + ".convert_to_arrow")
-    logger.debug("!!!!Using convert to arrow!!!")
     standard = {"DATE": pa.timestamp("ms")}
     if "DATE" in frame.columns:
         frame["DATE"] = pd.to_datetime(frame["DATE"])
@@ -34,7 +32,6 @@ def convert_to_arrow(frame):
             scheme.append(
                 (column_name, standard.get(column_name, pa.float32()))
             )
-    logger.debug(scheme)
     table = pa.Table.from_pandas(frame, schema=pa.schema(scheme))
     return table
 
@@ -70,11 +67,8 @@ def find_functions_and_docstring(submod):
     Returns:
         dictionary: includes functions and doc string
     """
-    logger = logging.getLogger(__file__ + ".find_func_and_info")
-
     import_path = "res2df." + submod
     func = importlib.import_module(import_path).df
-    logger.debug("Assigning %s to %s", func.__name__, submod)
     returns = {
         "extract": func,
         "options": tuple(
@@ -94,8 +88,6 @@ def _define_submodules():
     Returns:
         list: list of submodules
     """
-
-    logger = logging.getLogger(__file__ + "define_submodules")
     package_path = Path(res2df.__file__).parent
 
     submodules = {}
@@ -111,16 +103,8 @@ def _define_submodules():
             submod = "vfp"
         try:
             submodules[submod] = find_functions_and_docstring(submod_string)
-            logger.debug("Assigning %s to %s", submodules[submod], submod)
         except AttributeError:
-            logger.debug("No df function in %s", submod_path)
-
-    logger.debug(
-        "Returning the submodule names as a list: %s ", submodules.keys()
-    )
-    logger.debug(
-        "Returning the submodules extra args as a dictionary: %s ", submodules
-    )
+            pass  # No df function in submod_path, skip it
 
     return tuple(submodules.keys()), submodules
 
@@ -135,15 +119,14 @@ def find_md_log(submod, options):
     Returns:
         str|None: whatever contained in md_log_file
     """
-    logger = logging.getLogger(__file__ + ".find_md_log")
     if submod != "rft":
         return None
     # Special treatment of argument md_log_file
     md_log_file = options.get("md_log_file", None)
     try:
         del options["md_log_file"]
     except KeyError:
-        logger.debug("No md log provided")
+        pass  # No md log provided
 
     return md_log_file
 
@@ -219,9 +202,6 @@ def add_md_to_rft(rft_table, md_file_path):
     Returns:
         pd.Dataframe: the merged results
     """
-    logger = logging.getLogger(__file__ + ".add_md_to_rft")
-    logger.debug("Head of rft table prior to merge:\n %s", rft_table.head())
-
     try:
         md_table = pd.read_csv(md_file_path)
     except FileNotFoundError as fnfe:
@@ -235,19 +215,7 @@ def add_md_to_rft(rft_table, md_file_path):
     md_table[xtgeo_index_names] += 1
     md_table[xtgeo_index_names] = md_table[xtgeo_index_names].astype(int)
     xtgeo_to_rft_names = dict(zip(xtgeo_index_names, rft_index_names))
-    logger.debug(
-        "Datatypes, md_table: %s, rft_table: %s",
-        md_table[xtgeo_index_names].dtypes,
-        rft_table[rft_index_names].dtypes,
-    )
-    logger.debug(
-        "Shapes before merge rft: %s, md: %s", rft_table.shape, md_table.shape
-    )
     md_table.rename(xtgeo_to_rft_names, axis=1, inplace=True)
-    logger.debug("Header of md table after rename %s", md_table.head())
     rft_table = pd.merge(rft_table, md_table, on=rft_index_names, how="left")
-    logger.debug("Shape after merge %s", rft_table.shape)
-    logger.debug("Shape with no nans %s", rft_table.dropna().shape)
-    logger.debug("Head of merged table to return:\n %s", rft_table.head())
 
     return rft_table