#2172 - update hermes json files #2173

miroslavpojer · 2023-02-16T12:04:40Z

updated json files
repair of typo in menas password configuration

New hermes json files can be tested with this Hermes PR

- updated json files - repiar of typo in menas password

benedeki · 2023-02-21T10:40:59Z

examples/data/e2e_tests/test_jsons/castingConformanceRuleTest.json

-      "spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
-      "enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
-      "credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
+      "spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",


I think rest-api is also in the default resource files. Shouldn't it be consistent? 🤔

Sure. "_war" removed from all files in commit 314a6b2.

benedeki · 2023-02-21T10:41:26Z

examples/data/e2e_tests/test_jsons/castingConformanceRuleTest.json

-      "credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
+      "spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
+      "enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",
+      "credentials": "--rest-api-credentials-file ~/.ssh/menas-credential.properties",


Why this change? Seems like personal setup.

I would like to see it as small improvement to reach same pattern where CamelCase is used for Class files only. I am used to write these non-class files in format "word-word" or "word_word". I can see more similar locations with this naming. Are we following some format rules when to use CamelCase, w-w and w_w?

benedeki · 2023-02-21T10:42:34Z

examples/data/e2e_tests/test_jsons/coalesceConformanceRuleTest.json

-      "enceladus-job-jar": "spark-jobs/target/spark-jobs-2.19.0-SNAPSHOT.jar",
-      "credentials": "--rest-api-credentials-file ~/.ssh/menasCredential.properties",
+      "spark-conf": "--conf 'spark.driver.extraJavaOptions=-Denceladus.rest.uri=http://localhost:8080/rest_api_war/api -Dspline.mongodb.name=spline -Dspline.mongodb.url=mongodb://127.0.0.1:27017/ -Denceladus.recordId.generation.strategy=stableHashId'",
+      "enceladus-job-jar": "spark-jobs/target/spark-jobs-3.0.0-SNAPSHOT.jar",


Wonder if this somehow could be integrated with the current version. 🤔
This way it will be behind next month.

Added as point for improvement in Issue #2168

- Removed "_war" from rest_api uri definition.

benedeki

code reviewed
pulled
built
run

- Add properties to control 'datetimeRebaseModeInRead' for tests with ancient dates.

- Removed spline configuration properties.

…files' into feature/2172-update-hermes-json-files # Conflicts: # examples/data/e2e_tests/test_jsons/stdNfDnTest.json # examples/data/e2e_tests/test_jsons/stdNfDyTest.json # examples/data/e2e_tests/test_jsons/stdNtDnTest.json # examples/data/e2e_tests/test_jsons/stdNtDyTest.json

dk1844

@benedeki is right about the hardcoded version problem.

Otherwise LGTM (just read the code)

dk1844 · 2023-02-28T10:16:25Z

rest-api/src/test/resources/application.properties

@@ -57,7 +57,7 @@ [email protected]
 enceladus.rest.hadoop.auth.krb5.keytab=hdfs.keytab

 enceladus.rest.auth.inmemory.user=user
-enceladus.rest.auth.inmemory.password=chang< )eme
+enceladus.rest.auth.inmemory.password=changeme


Are you sure this was a typo?
Could it have been a way to force the user of hermes to provide their own application.properties while this file serving only as an example?

You are right. After some searching I have found answer from author about goal of this "typo". Reverted in commit 0f21273.

- Return to origin. Found record in history that it is not a typo but value for testing.

dk1844

LGTM (just read the code)

Zejnilovic · 2023-03-03T08:16:08Z

examples/data/e2e_tests/test_jsons/stdNtDnTest.json

@@ -30,7 +30,7 @@
        "pluginName" : "DatasetComparison",
        "name": "DatasetComparison",
        "order" : 1,
-        "args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "name" ],
+        "args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property", "--datetimeRebaseMode", "LEGACY"],


Suggested change

"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property", "--datetimeRebaseMode", "LEGACY"],

"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property"],

I think we said this is not needed anymore

We spoke about it, but my experiments confirmed that for four std json files this settings needs to be there.

So we updated Hermes to include datetimeRebaseMode?

We removed the opting afaik - AbsaOSS/hermes#132

You are right, I was confused. datetimeRebaseMode removed in commit 92d983d.

- Removed no more needed parameter for DatasetComparison

benedeki

My comments were addressed, seems the ones of others too...

dk1844

LGTM (just read the code)

- Update version to fix server build.

…files' into feature/2172-update-hermes-json-files

menas/ui/package.json

Revert version to correct one.

sonarcloud · 2023-03-22T09:13:28Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

dk1844

LGTM (I am sorry you had to switch Enceladus version back and forth -- not my intention indeed to complicate related setup)

#2172 - update hermes json files

8b89100

- updated json files - repiar of typo in menas password

miroslavpojer requested review from benedeki, dk1844, lsulak and jakipatryk as code owners February 16, 2023 12:04

miroslavpojer self-assigned this Feb 16, 2023

miroslavpojer added the tests QA/test-specific label Feb 16, 2023

benedeki reviewed Feb 21, 2023

View reviewed changes

#2172 - update hermes json files

314a6b2

- Removed "_war" from rest_api uri definition.

benedeki previously approved these changes Feb 23, 2023

View reviewed changes

Merge branch 'develop' into feature/2172-update-hermes-json-files

007dd05

miroslavpojer added PR:no testing needed Only for PR - PR doesn't need to be tested by a tester (person) and removed tests QA/test-specific labels Feb 23, 2023

#2172 - update hermes json files

2d3d228

- Add properties to control 'datetimeRebaseModeInRead' for tests with ancient dates.

miroslavpojer dismissed benedeki’s stale review via 2d3d228 February 24, 2023 09:20

miroslavpojer added 2 commits February 27, 2023 11:57

#2172 - update hermes json files

7c9486d

- Removed spline configuration properties.

Zejnilovic previously approved these changes Feb 27, 2023

View reviewed changes

dk1844 previously approved these changes Feb 28, 2023

View reviewed changes

#2172 - update hermes json files

0f21273

- Return to origin. Found record in history that it is not a typo but value for testing.

miroslavpojer dismissed stale reviews from dk1844 and Zejnilovic via 0f21273 February 28, 2023 11:46

miroslavpojer requested review from dk1844 and Zejnilovic February 28, 2023 11:49

dk1844 previously approved these changes Mar 1, 2023

View reviewed changes

Zejnilovic reviewed Mar 3, 2023

View reviewed changes

miroslavpojer requested review from Zejnilovic and dk1844 March 15, 2023 08:11

#2172 - update hermes json files

92d983d

- Removed no more needed parameter for DatasetComparison

miroslavpojer dismissed dk1844’s stale review via 92d983d March 16, 2023 10:34

benedeki previously approved these changes Mar 16, 2023

View reviewed changes

Merge branch 'develop' into feature/2172-update-hermes-json-files

25d9728

dk1844 previously approved these changes Mar 21, 2023

View reviewed changes

miroslavpojer and others added 4 commits March 21, 2023 14:27

Merge branch 'develop' into feature/2172-update-hermes-json-files

0f005fe

Merge branch 'develop' into feature/2172-update-hermes-json-files

125af38

#2172 - update hermes json files

9b97eb2

- Update version to fix server build.

Merge remote-tracking branch 'origin/feature/2172-update-hermes-json-…

726d29a

…files' into feature/2172-update-hermes-json-files

miroslavpojer dismissed stale reviews from dk1844 and benedeki via 726d29a March 21, 2023 15:51

miroslavpojer commented Mar 22, 2023

View reviewed changes

menas/ui/package.json Outdated Show resolved Hide resolved

Update menas/ui/package.json

95a355c

Revert version to correct one.

miroslavpojer requested review from dk1844 and benedeki March 22, 2023 10:08

dk1844 approved these changes Mar 22, 2023

View reviewed changes

miroslavpojer merged commit d2aa209 into develop Mar 22, 2023

miroslavpojer deleted the feature/2172-update-hermes-json-files branch March 22, 2023 10:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#2172 - update hermes json files #2173

#2172 - update hermes json files #2173

miroslavpojer commented Feb 16, 2023 •

edited

Loading

benedeki Feb 21, 2023

miroslavpojer Feb 22, 2023

benedeki Feb 21, 2023

miroslavpojer Feb 22, 2023

benedeki Feb 21, 2023

miroslavpojer Feb 22, 2023

benedeki left a comment

dk1844 left a comment

dk1844 Feb 28, 2023

miroslavpojer Feb 28, 2023

dk1844 left a comment

Zejnilovic Mar 3, 2023

miroslavpojer Mar 13, 2023

Zejnilovic Mar 15, 2023

miroslavpojer Mar 16, 2023

benedeki left a comment

dk1844 left a comment

sonarcloud bot commented Mar 22, 2023

dk1844 left a comment

	"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property", "--datetimeRebaseMode", "LEGACY"],
	"args" : ["--format", "parquet", "--new-path", "#{new-std-data-path}#", "--ref-path", "#{ref-std-data-path}#", "--keys", "property"],

#2172 - update hermes json files #2173

#2172 - update hermes json files #2173

Conversation

miroslavpojer commented Feb 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benedeki left a comment

Choose a reason for hiding this comment

dk1844 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dk1844 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benedeki left a comment

Choose a reason for hiding this comment

dk1844 left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Mar 22, 2023

dk1844 left a comment

Choose a reason for hiding this comment

miroslavpojer commented Feb 16, 2023 •

edited

Loading