Invoke validations from import executor #1141
Conversation
vish-cs
force-pushed
the
executor
branch
3 times, most recently
from
70e507b to
bc84404
Compare
| # Invoke validations before upload. | ||
| invoke_import_validation: bool = True | ||
| # Import validation config file. | ||
| validation_config_file: str = 'tools/validation/config.json' |
There was a problem hiding this comment.
can we call this validation_config.json?
There are a lot of config.json files in the repo.
| mcf_path) + '.old' | ||
| differ_results_path = os.path.join(absolute_import_dir, 'results') | ||
| config_file_path = os.path.join(absolute_import_dir, | ||
| self.config.validation_config_file) |
There was a problem hiding this comment.
Can this be an import specific validation config declared in the manifest.json and default to config.validation_config_file if not declared?
There was a problem hiding this comment.
Yes I have added a conditional assignment to check for it in the import spec first, else use the default value from the config
| previous_data_path, | ||
| '--output_location=' + | ||
| differ_results_path) | ||
| process = _run_user_script( |
There was a problem hiding this comment.
Can we call the differ module directly instead of a process?
| ) | ||
| else: | ||
| # Run import script locally. | ||
| script_interpreter = _get_script_interpreter( |
There was a problem hiding this comment.
Pls retain this as the interpreter path changes for imports using shell scripts
There was a problem hiding this comment.
This logic is not removed..it's just refactored out into a separate function
| @@ -111,6 +111,12 @@ class ExecutorConfig: | |||
| user_script_args: List[str] = () | |||
| # Environment variables for the user script | |||
| user_script_env: dict = None | |||
| # Skip uploading the data to GCS (for local testing). | |||
| skip_gcs_upload: bool = True | |||
There was a problem hiding this comment.
Should we turn this to False in the config_overrides.json?
There was a problem hiding this comment.
I have removed this flag from this PR
| # Skip uploading the data to GCS (for local testing). | ||
| skip_gcs_upload: bool = True | ||
| # Invoke validations before upload. | ||
| invoke_import_validation: bool = True |
There was a problem hiding this comment.
Let's keep this as False until we add call to dc-import so the previous version of MCF is stored in GCS
No description provided.