Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#2172 - update hermes json files #2173

Merged
merged 14 commits into from
Mar 22, 2023
Merged

#2172 - update hermes json files #2173

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
8b89100
#2172 - update hermes json files
miroslavpojer Feb 16, 2023
314a6b2
#2172 - update hermes json files
miroslavpojer Feb 22, 2023
007dd05
Merge branch 'develop' into feature/2172-update-hermes-json-files
miroslavpojer Feb 23, 2023
2d3d228
#2172 - update hermes json files
miroslavpojer Feb 24, 2023
7c9486d
#2172 - update hermes json files
miroslavpojer Feb 27, 2023
b2e01ca
Merge remote-tracking branch 'origin/feature/2172-update-hermes-json-…
miroslavpojer Feb 27, 2023
0f21273
#2172 - update hermes json files
miroslavpojer Feb 28, 2023
92d983d
#2172 - update hermes json files
miroslavpojer Mar 16, 2023
25d9728
Merge branch 'develop' into feature/2172-update-hermes-json-files
miroslavpojer Mar 21, 2023
0f005fe
Merge branch 'develop' into feature/2172-update-hermes-json-files
miroslavpojer Mar 21, 2023
125af38
Merge branch 'develop' into feature/2172-update-hermes-json-files
miroslavpojer Mar 21, 2023
9b97eb2
#2172 - update hermes json files
miroslavpojer Mar 21, 2023
726d29a
Merge remote-tracking branch 'origin/feature/2172-update-hermes-json-…
miroslavpojer Mar 21, 2023
95a355c
Update menas/ui/package.json
miroslavpojer Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions examples/data/e2e_tests/test_jsons/castingConformanceRuleTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think rest-api is also in the default resource files. Shouldn't it be consistent? 🤔

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. "_war" removed from all files in commit 314a6b2.

"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change? Seems like personal setup.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to see it as small improvement to reach same pattern where CamelCase is used for Class files only. I am used to write these non-class files in format "word-word" or "word_word". I can see more similar locations with this naming. Are we following some format rules when to use CamelCase, w-w and w_w?

"ref-std-data-path": "/ref/castingConformanceRule/std",
"new-std-data-path": "/tmp/conformance-output/standardized-castingConformanceRule-1-2020-03-23-1",
"ref-publish-data-path": "/ref/castingConformanceRule/publish",
Expand Down
6 changes: 3 additions & 3 deletions examples/data/e2e_tests/test_jsons/coalesceConformanceRuleTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder if this somehow could be integrated with the current version. 🤔
This way it will be behind next month.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added as point for improvement in Issue #2168

"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/coalesceConformanceRule/std",
"new-std-data-path": "/tmp/conformance-output/standardized-coalesceConformanceRule-1-2020-03-23-1",
"ref-publish-data-path": "/ref/coalesceConformanceRule/publish",
Expand Down
6 changes: 3 additions & 3 deletions examples/data/e2e_tests/test_jsons/concatenationConformanceRuleTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/concatenationConformanceRule/std",
"new-std-data-path": "/tmp/conformance-output/standardized-concatenationConformanceRule-1-2020-03-23-1",
"ref-publish-data-path": "/ref/concatenationConformanceRule/publish",
Expand Down
6 changes: 3 additions & 3 deletions examples/data/e2e_tests/test_jsons/literalConformanceRuleTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/literalConformanceRule/std",
"new-std-data-path": "/tmp/conformance-output/standardized-literalConformanceRule-1-2020-03-23-1",
"ref-publish-data-path": "/ref/literalConformanceRule/publish",
Expand Down
6 changes: 3 additions & 3 deletions examples/data/e2e_tests/test_jsons/mappingConformanceRuleTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/mappingConformanceRule/std",
"new-std-data-path": "/tmp/conformance-output/standardized-mappingConformanceRule-1-2020-03-23-1",
"ref-publish-data-path": "/ref/mappingConformanceRule/publish",
Expand Down
6 changes: 3 additions & 3 deletions examples/data/e2e_tests/test_jsons/negationConformanceRuleTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/negationConformanceRule/std",
"new-std-data-path": "/tmp/conformance-output/standardized-negationConformanceRule-1-2020-03-23-1",
"ref-publish-data-path": "/ref/negationConformanceRule/publish",
Expand Down
6 changes: 3 additions & 3 deletions examples/data/e2e_tests/test_jsons/singleColumnConformanceRuleTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/singleColumnConformanceRule/std",
"new-std-data-path": "/tmp/conformance-output/standardized-singleColumnConformanceRule-1-2020-03-23-1",
"ref-publish-data-path": "/ref/singleColumnConformanceRule/publish",
Expand Down
10 changes: 5 additions & 5 deletions examples/data/e2e_tests/test_jsons/stdNfDnTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/std_nf_dn/std",
"new-std-data-path": "/tmp/conformance-output/standardized-std_nf_dn-1-2019-11-27-1",
"results-log-path": "/std/std_nf_dn/results",
Expand All @@ -15,7 +15,7 @@
"pluginName" : "BashPlugin",
"name": "Standardization",
"order" : 0,
"args" : ["#{spark-submit}# #{spark-conf}# --class za.co.absa.enceladus.standardization.StandardizationJob #{enceladus-job-jar}# #{credentials}# #{dataset}# --raw-format csv --header true "],
"args" : ["#{spark-submit}# #{spark-conf}# --class za.co.absa.enceladus.standardization.StandardizationJob #{enceladus-job-jar}# #{credentials}# #{dataset}# --raw-format json"],
"writeArgs": []
},
{
Expand All @@ -30,7 +30,7 @@
"pluginName" : "DatasetComparison",
"name": "DatasetComparison",
"order" : 1,
"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "name" ],
"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property", "--datetimeRebaseMode", "LEGACY"],
"writeArgs": ["--out-path", "#{results-log-path}#/stdDataDiff"],
"dependsOn": "Standardization"
}
Expand Down
10 changes: 5 additions & 5 deletions examples/data/e2e_tests/test_jsons/stdNfDyTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/std_nf_dy/std",
"new-std-data-path": "/tmp/conformance-output/standardized-std_nf_dy-1-2019-11-27-1",
"results-log-path": "/std/std_nf_dy/results",
Expand All @@ -15,7 +15,7 @@
"pluginName" : "BashPlugin",
"name": "Standardization",
"order" : 0,
"args" : ["#{spark-submit}# #{spark-conf}# --class za.co.absa.enceladus.standardization.StandardizationJob #{enceladus-job-jar}# #{credentials}# #{dataset}# --raw-format csv --header true "],
"args" : ["#{spark-submit}# #{spark-conf}# --class za.co.absa.enceladus.standardization.StandardizationJob #{enceladus-job-jar}# #{credentials}# #{dataset}# --raw-format json"],
"writeArgs": []
},
{
Expand All @@ -30,7 +30,7 @@
"pluginName" : "DatasetComparison",
"name": "DatasetComparison",
"order" : 1,
"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "name" ],
"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property", "--datetimeRebaseMode", "LEGACY"],
"writeArgs": ["--out-path", "#{results-log-path}#/stdDataDiff"],
"dependsOn": "Standardization"
}
Expand Down
10 changes: 5 additions & 5 deletions examples/data/e2e_tests/test_jsons/stdNtDnTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/std_nt_dn/std",
"new-std-data-path": "/tmp/conformance-output/standardized-std_nt_dn-1-2019-11-27-1",
"results-log-path": "/std/std_nt_dn/results",
Expand All @@ -15,7 +15,7 @@
"pluginName" : "BashPlugin",
"name": "Standardization",
"order" : 0,
"args" : ["#{spark-submit}# #{spark-conf}# --class za.co.absa.enceladus.standardization.StandardizationJob #{enceladus-job-jar}# #{credentials}# #{dataset}# --raw-format csv --header true "],
"args" : ["#{spark-submit}# #{spark-conf}# --class za.co.absa.enceladus.standardization.StandardizationJob #{enceladus-job-jar}# #{credentials}# #{dataset}# --raw-format json"],
"writeArgs": []
},
{
Expand All @@ -30,7 +30,7 @@
"pluginName" : "DatasetComparison",
"name": "DatasetComparison",
"order" : 1,
"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "name" ],
"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property", "--datetimeRebaseMode", "LEGACY"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property", "--datetimeRebaseMode", "LEGACY"],
"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property"],

I think we said this is not needed anymore

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We spoke about it, but my experiments confirmed that for four std json files this settings needs to be there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we updated Hermes to include datetimeRebaseMode?

We removed the opting afaik - AbsaOSS/hermes#132

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, I was confused. datetimeRebaseMode removed in commit 92d983d.

"writeArgs": ["--out-path", "#{results-log-path}#/stdDataDiff"],
"dependsOn": "Standardization"
}
Expand Down
10 changes: 5 additions & 5 deletions examples/data/e2e_tests/test_jsons/stdNtDyTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/std_nt_dy/std",
"new-std-data-path": "/tmp/conformance-output/standardized-std_nt_dy-1-2019-11-27-1",
"results-log-path": "/std/std_nt_dy/results",
Expand All @@ -15,7 +15,7 @@
"pluginName" : "BashPlugin",
"name": "Standardization",
"order" : 0,
"args" : ["#{spark-submit}# #{spark-conf}# --class za.co.absa.enceladus.standardization.StandardizationJob #{enceladus-job-jar}# #{credentials}# #{dataset}# --raw-format csv --header true "],
"args" : ["#{spark-submit}# #{spark-conf}# --class za.co.absa.enceladus.standardization.StandardizationJob #{enceladus-job-jar}# #{credentials}# #{dataset}# --raw-format json"],
"writeArgs": []
},
{
Expand All @@ -30,7 +30,7 @@
"pluginName" : "DatasetComparison",
"name": "DatasetComparison",
"order" : 1,
"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "name" ],
"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property", "--datetimeRebaseMode", "LEGACY"],
"writeArgs": ["--out-path", "#{results-log-path}#/stdDataDiff"],
"dependsOn": "Standardization"
}
Expand Down
6 changes: 3 additions & 3 deletions examples/data/e2e_tests/test_jsons/uppercaseConformanceRuleTest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
{
"vars": {
"spark-submit": "spark-submit --num-executors 2 --executor-memory 2G --deploy-mode client",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
"spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
"enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
"credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",
"ref-std-data-path": "/ref/uppercaseConformanceRule/std",
"new-std-data-path": "/tmp/conformance-output/standardized-uppercaseConformanceRule-1-2020-03-23-1",
"ref-publish-data-path": "/ref/uppercaseConformanceRule/publish",
Expand Down
2 changes: 1 addition & 1 deletion rest-api/src/test/resources/application.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ [email protected]
enceladus.rest.hadoop.auth.krb5.keytab=hdfs.keytab

enceladus.rest.auth.inmemory.user=user
enceladus.rest.auth.inmemory.password=chang< )eme
enceladus.rest.auth.inmemory.password=changeme
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure this was a typo?
Could it have been a way to force the user of hermes to provide their own application.properties while this file serving only as an example?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. After some searching I have found answer from author about goal of this "typo". Reverted in commit 0f21273.

enceladus.rest.auth.inmemory.admin.user=menas_admin
enceladus.rest.auth.inmemory.admin.password=admin123

Expand Down