Feature eng dev mmt #157
Feature eng dev mmt #157
Conversation
…e_alphanumeric_underscore` .}}/feature_engineering/notebooks/GenerateAndWriteFeatures.py.tmpl
…ing code snippets
…ing code snippets
| @@ -103,7 +103,7 @@ raw_data = spark.read.format("delta").load(input_table_path) | |||
| # Compute the features. This is done by dynamically loading the features module. | |||
| from importlib import import_module | |||
|
|
|||
| mod = import_module(features_module) | |||
| mod = import_module(features_module) ##?? | |||
There was a problem hiding this comment.
What is this comment for?
There was a problem hiding this comment.
Good catch! Should have been omitted -- maybe I deleted after the commit -- will recommit
|
|
||
| fs = feature_store.FeatureStoreClient() | ||
|
|
||
| from databricks.feature_engineering import FeatureEngineeringClient |
There was a problem hiding this comment.
We should only do this if UC is enabled, you can see other examples where we use:
{{ if (eq .input_include_models_in_unity_catalog `no`) }}, or does this client work for non-UC cases as well?
There was a problem hiding this comment.
OK -- I think need to update with conditionals. I was wondering where they were but they need to be added!
…eering related notebooks
| @@ -116,11 +116,11 @@ features_df = compute_features_fn( | |||
| # COMMAND ---------- | |||
| # DBTITLE 1, Write computed features. | |||
|
|
|||
| {{- if (eq .input_include_models_in_unity_catalog "no") }} | |||
There was a problem hiding this comment.
Is there a reason you changed this from your last commit?
| {{- if (eq .input_include_models_in_unity_catalog "no") }} | |
| {{- if (eq .input_include_models_in_unity_catalog `no`) }} |
There was a problem hiding this comment.
There were some failed tests and I noted previous conditionals in section above related to dbutils.wigets with {{- if (eq .input_include_models_in_unity_catalog "no") }} so thought to replace... should it be 'no' instead of "no" ?
There was a problem hiding this comment.
Hmm generally we use backticks (`) instead of quotes (") or ('). You can see other examples here: https://github.com/search?q=repo%3Adatabricks%2Fmlops-stacks%20input_include_models_in_unity_catalog%20&type=code
There was a problem hiding this comment.
It's inconsistent within the existing repo. I will use backticks in my updates :)
| label="fare_amount", | ||
| exclude_columns=exclude_columns, | ||
| ) | ||
| {{end -}} |
There was a problem hiding this comment.
So the - at the end means to remove space after this line. You should try generating this notebook in your local Stacks repo by using databricks bundle init and add screenshots of both UC and non-UC to make sure they look as expected
There was a problem hiding this comment.
Will try generating -- I didn't realize I could! Thanks
|
@hengrumay FYI only two tests seem to be failing now and they're about links. We can wait once you're ready to merge and then fix broken links together |
…nference/predict.py.tmpl
|
@arpitjasa-db: updated |
| {{ if (eq .input_include_feature_store `yes`) }} | ||
| from databricks.feature_store import FeatureStoreClient | ||
|
|
||
| {{ if (eq .input_include_feature_store `yes`) }} |
There was a problem hiding this comment.
see latest updates -- will be adding links to screenshots
| output_df = ( | ||
| prediction_df.withColumn("prediction", prediction_df["prediction"]) | ||
| .withColumn("model_version", lit(model_version)) | ||
| .withColumn("inference_timestamp", to_timestamp(lit(ts))) | ||
| ) | ||
| {{ else }} | ||
| {{ else }} |
There was a problem hiding this comment.
see latest updates -- will be adding links to screenshots
| @@ -233,6 +264,30 @@ training_set = fs.create_training_set( | |||
| exclude_columns=exclude_columns, | |||
| ) | |||
|
|
|||
| {{else}} | |||
There was a problem hiding this comment.
see latest updates -- will be adding links to screenshots
|
Well that was 'unexpected' wrt passed checks lol @arpitjasa-db FYI: Screenshots for recent local init and review updates: |
Updating code with feature store (fs) to feature engineering(fe) wrt https://databricks.atlassian.net/browse/MLOPS-675