Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

generate shorter pseudo-random IDs with better spread than current time #1122

Open
benclifford opened this issue Jul 8, 2019 · 4 comments
Open

generate shorter pseudo-random IDs with better spread than current time #1122

benclifford opened this issue Jul 8, 2019 · 4 comments
Labels
outreachy Good initial contributions for Outreachy applicants

Comments

@benclifford
Copy link
Collaborator

There are lots of places in the code where unique IDs are generated mostly based on the current time. That's a big ID space but with lots of correlation between IDs generated by the same run. At some scale these will clash and cause non-deterministic trouble that will be hard to debug.

There is plenty of experience in the world using more spread IDs (c.f. UUIDs)

Everywhere time() is used to generate unique IDs should be replaced by a call to a parsl helper function that does this more sensibly.
@benclifford benclifford mentioned this issue Jul 11, 2019
Merged
@benclifford benclifford changed the title generate pseudo-random IDs with better spread than current time generate shorter pseudo-random IDs with better spread than current time Aug 29, 2019
@benclifford
Copy link
Collaborator Author

crossref #831

@benclifford benclifford mentioned this issue Sep 26, 2019
Open
@benclifford benclifford mentioned this issue Oct 1, 2021
Closed
@benclifford benclifford mentioned this issue May 12, 2023
Closed
@benclifford
Copy link
Collaborator Author

A name such as parsl.localprovider.1684151141.6619916.sh.out contains around 56 bits of potentially distinguishing information: 17 digits * log_2 10 bits per digit.

(noting here since I just calculated it for other purposes)

benclifford added a commit that referenced this issue May 15, 2023
@benclifford
Put executor name and block ID in provider submit script filenames
460d181 
Prior to this PR, submit script filenames looked like this:

parsl.localprovider.1684151066.651144.sh

After this PR, submit script filenames look like this:

parsl.htex_Local.block-0.1684154520.747874.sh

This PR fixes feature request issue #2692, which requests that the
filename contain more block identifying context, for correlation with
other block-related log files.

Also adds some more uniqueness which is relevant to issue #1122 and
addresses the naming collisions I have experienced in that context.
@benclifford benclifford mentioned this issue May 15, 2023
Merged
benclifford added a commit that referenced this issue May 25, 2023
@benclifford
Put executor name and block ID in provider submit script filenames (#…
2d748ef 
…2710)

Prior to this PR, submit script filenames looked like this:

parsl.localprovider.1684151066.651144.sh

After this PR, submit script filenames look like this:

parsl.htex_Local.block-0.1684154520.747874.sh

This PR fixes feature request issue #2692, which requests that the filename contain more block identifying context, for correlation with other block-related log files.

Also adds some more uniqueness which is relevant to issue #1122 and addresses the naming collisions I have experienced in that context.
@benclifford benclifford added the outreachy Good initial contributions for Outreachy applicants label Mar 13, 2024
@OfficialOzioma
Copy link

Hi @benclifford is this issue still open, i will like to work on it, and i will need some guidance on how to get started thank you

@benclifford benclifford mentioned this issue Aug 19, 2024
Merged
@rjmello
Copy link
Member

rjmello commented Oct 21, 2024

Crossref: #3639

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
outreachy Good initial contributions for Outreachy applicants
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@benclifford @OfficialOzioma @rjmello