Slow downloads from CivitAI

[Lalimec]

This is probably due to civitai api but I wanted to know if you have some idea about the issue. A basic wget downloads much faster than using this Model Manager or Workspace Manager (another custom node that offers civitai download functions) etc. I was wondering if anything can be done to increase the download speeds? I am also using an ec2 instance, not sure if that affects anything though.

[cdb-boop]

Not sure. Could be python slowness, something to do with multithreading or a code mistake. The relevant code is here. It might be possible to have python make a wget call to the system, but it also might require OS specific code and error handling.

If you want to test if it's an issue on Civitai's end, you could try downloading a model that is also on HuggingFace, since it also should be supported.

I personally haven't noticed that big of a difference as compared to downloading in a web browser, but if you could give some timings that show a significant slowdown, then I would be willing to look into solutions.

[Lalimec]

Turns out its not just civitai. Model manager doesnt show speed so ive taken two screenshots to compare the speeds. Yes they are downloading at the same time so there is going to be a slow down obv but you get my point. I remember back in the day the dev of CIvitai Browser fixed this by using aria or sth in a1111.

Also is there a way to cancel a download that im not seeing?

[sansmoraxz]

Check #36.

Also, if anyone has figured out proper way to do multipart downloads form civitai that may help improve this more. Namely they block HEAD requests and the size returned by API call is in floating point kB. Multiplying by 1024 may work but seems risky.

Ref article for multipart download:
https://hackernoon.com/how-to-speed-up-file-downloads-with-python