Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BACK-43] jellyfish updates based on reuse of legacy _id for migration #203

Open
wants to merge 24 commits into
base: master
Choose a base branch
from

Conversation

jh-bate
Copy link
Contributor

@jh-bate jh-bate commented May 29, 2024

  • include _id as the _deduplicator hash for use in the platform deduplicator code
  • set the upload records to include the "legacy" _deduplicator name and version for use in the platform deduplicator code
  • include up loader destination for switch from jellyfish to platform API during migration

see tidepool-org/development#297

@jh-bate jh-bate changed the title jellyfish updates based on reuse of legacy _id for migration [BACK-43] jellyfish updates based on reuse of legacy _id for migration Jun 24, 2024
@jh-bate
Copy link
Contributor Author

jh-bate commented Jun 24, 2024

/deploy qa3

@tidebot
Copy link
Collaborator

tidebot commented Jun 24, 2024

jh-bate updated values.yaml file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Jun 24, 2024

jh-bate updated flux policies file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Jun 24, 2024

jh-bate deployed jellyfish add-dedup-hash branch to qa3 namespace

@jh-bate
Copy link
Contributor Author

jh-bate commented Aug 1, 2024

/deploy qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 1, 2024

jh-bate updated values.yaml file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 1, 2024

jh-bate updated flux policies file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 1, 2024

jh-bate deployed jellyfish add-dedup-hash branch to qa3 namespace

@jh-bate
Copy link
Contributor Author

jh-bate commented Aug 5, 2024

/deploy qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 5, 2024

jh-bate updated values.yaml file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 5, 2024

jh-bate updated flux policies file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 5, 2024

jh-bate deployed jellyfish add-dedup-hash branch to qa3 namespace

@jh-bate
Copy link
Contributor Author

jh-bate commented Aug 5, 2024

/deploy qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 5, 2024

jh-bate updated values.yaml file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 5, 2024

jh-bate updated flux policies file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 5, 2024

jh-bate deployed jellyfish add-dedup-hash branch to qa3 namespace

@jh-bate
Copy link
Contributor Author

jh-bate commented Aug 6, 2024

/deploy qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 6, 2024

jh-bate updated values.yaml file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 6, 2024

jh-bate updated flux policies file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 6, 2024

jh-bate deployed jellyfish add-dedup-hash branch to qa3 namespace

@jh-bate
Copy link
Contributor Author

jh-bate commented Aug 6, 2024

/deploy qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 6, 2024

jh-bate updated values.yaml file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 6, 2024

jh-bate updated flux policies file in qa3

@tidebot
Copy link
Collaborator

tidebot commented Aug 6, 2024

jh-bate deployed jellyfish add-dedup-hash branch to qa3 namespace

@jh-bate jh-bate mentioned this pull request Sep 17, 2024
Copy link
Contributor

@darinkrauss darinkrauss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps a few changes. What do you think?

lib/schema/schema.js Outdated Show resolved Hide resolved
lib/misc.js Show resolved Hide resolved
lib/misc.js Outdated Show resolved Hide resolved
env.js Outdated Show resolved Hide resolved
lib/jellyfishService.js Outdated Show resolved Hide resolved
lib/jellyfishService.js Outdated Show resolved Hide resolved
lib/jellyfishService.js Outdated Show resolved Hide resolved
lib/jellyfishService.js Outdated Show resolved Hide resolved
Copy link
Contributor

@darinkrauss darinkrauss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question, one possible change. Let me know what you think and I can approve without changes.

env.js Outdated Show resolved Hide resolved
lib/jellyfishService.js Outdated Show resolved Hide resolved
darinkrauss
darinkrauss previously approved these changes Nov 2, 2024
Copy link
Contributor

@darinkrauss darinkrauss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One comment on a bit of simplification, if you so choose.

Also, there are a number of outstanding vulnerabilities (npm audit, snyk, dependabot). I ran my usual update process and created a new PR on top of this one. You can either merge into this PR or cherry pick it. The PR is #205.

lib/schema/schemaEnv.js Outdated Show resolved Hide resolved
darinkrauss
darinkrauss previously approved these changes Nov 10, 2024
Copy link
Contributor

@darinkrauss darinkrauss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! A couple of comments/questions, but nothing blocking approval.

@@ -0,0 +1 @@
[]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might as well go ahead and check in similar files for all the environments. That way if this is deployed anywhere it will still function, but effectively act the same and use Jellyfish.

lib/misc.js Outdated
Comment on lines 69 to 71
var iv = `${env}-environment`.substring(0, 16);
var key = serverSecret.substring(0, 32);
var algorithm = 'aes-256-cbc';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Consider pulling these out into a separate function so we guarantee that encryption and decryption use the same values.

lib/misc.js Outdated
var key = serverSecret.substring(0, 32);
var algorithm = 'aes-256-cbc';

var encryptedArray = dataArray.map((item) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, so you are encrypting each user id separately, not just the entire file?

lib/misc.js Outdated
@@ -65,3 +65,35 @@ exports.generateId = function(fields) {
return base32hex.encodeBuffer(hasher.digest(), { paddingChar: '-' });
};

exports.encryptArrayToFile = function (dataArray, filePath, env, serverSecret) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we use this when creating/updating the unencrypted file?

* userid hash for platform users
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants