Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelise fetchers #169

Open
Atemu opened this issue Mar 18, 2022 · 3 comments · Fixed by #175
Open

Parallelise fetchers #169

Atemu opened this issue Mar 18, 2022 · 3 comments · Fixed by #175

Comments

@Atemu
Copy link
Contributor

Atemu commented Mar 18, 2022

I just tried updating LOS but GH is painfully slow at serving the repos sparsely (one kernel source was in the KiB/s...).

The scripts need to be parallelised, especially now that we've got 3 "sub-flavours" to boot.

Is there an easy way to parallelise a for loop in python?
@Kranzes
Copy link
Member

Kranzes commented Mar 18, 2022

Is this an issue mk-repo or just the device/vendor fetchers?

@Atemu
Copy link
Contributor Author

Atemu commented Mar 18, 2022 •
edited
Loading

The metadata fetcher is so fast that it's not worth parallelising if I'm not mistaken. Though it'd be nice if the repo username check could be done away with.

The device/vendor dir updater is the bug issue.

zhaofengli added a commit to zhaofengli/robotnix that referenced this issue Mar 23, 2022
@zhaofengli
mk_repo_file: Parallelize fetch
b63dea4 
Fixes nix-community#169.
zhaofengli added a commit to zhaofengli/robotnix that referenced this issue Mar 23, 2022
@zhaofengli
mk_repo_file: Parallelize fetch
cac56ef 
Fixes nix-community#169.
zhaofengli added a commit to zhaofengli/robotnix that referenced this issue Mar 24, 2022
@zhaofengli
mk_repo_file: Parallelize fetch
102adae 
Fixes nix-community#169.
zhaofengli added a commit to zhaofengli/robotnix that referenced this issue Mar 24, 2022
@zhaofengli
mk_repo_file: Parallelize fetch
0b3bb05 
Fixes nix-community#169.
@zhaofengli zhaofengli mentioned this issue Mar 24, 2022
Merged
zhaofengli added a commit to zhaofengli/robotnix that referenced this issue Mar 24, 2022
@zhaofengli
mk_repo_file: Parallelize fetch
eda1dd0 
Fixes nix-community#169.
danielfullmer pushed a commit that referenced this issue Mar 25, 2022
@zhaofengli @danielfullmer
mk_repo_file: Parallelize fetch
fb09bc9 
Fixes #169.
@Atemu
Copy link
Contributor Author

Atemu commented Mar 28, 2022

Ah, just noticed #175 doesn't actually fix this issue as it "only" parallelises the metadata/repo fetch, not the updating of the LOS repos.

@danielfullmer danielfullmer reopened this Mar 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

mk_repo_file: Parallelize fetch zhaofengli/robotnix
3 participants
@danielfullmer @Atemu @Kranzes