Issue/dont save if nan (#71)

* drop nans if in generation_mw * pydantic<2.0 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
openclimatefix · Sep 5, 2023 · 8169936 · 8169936
1 parent 880ee11
commit 8169936
Show file tree

Hide file tree

Showing 2 changed files with 4 additions and 0 deletions.
diff --git a/gspconsumer/app.py b/gspconsumer/app.py
@@ -194,6 +194,9 @@ def pull_data_and_save(
             else:
                 logger.debug(f"This is the first lot gsp yield data for GSP {(gsp.gsp_id)}")
 
+            # drop any nan values in generation_mw column
+            gsp_yield_df = gsp_yield_df.dropna(subset=["generation_mw"])
+
             # need columns datetime_utc, solar_generation_kw
             gsp_yield_df["solar_generation_kw"] = 1000 * gsp_yield_df["generation_mw"]
             gsp_yield_df["datetime_utc"] = gsp_yield_df["datetime_gmt"]

diff --git a/requirements.txt b/requirements.txt
@@ -1,4 +1,5 @@
 numpy
+pydantic<2.0
 pandas
 pyyaml
 urllib3