Failed to download model!

[seadotline]

I tried downloading a few models, but got this message:

"Failed to download model!

Unable to download from Civitai! Redirect url: /login?returnUrl=%2Fmodel-versions%2F297334&reason=download-auth"

[cdb-boop]

Yes, I encountered this yesterday. I think I fixed it on my dev fork. I think it only affects content above the PG rating.

[seadotline]

Thanks.
I tried using dev fork, but unfortunately the problem is still here.

I think it's not only the model rating, but as far as I know, authors can close downloads without authorization.

For example, I can't download this model.
https://civitai.com/models/7371?modelVersionId=425083

[cdb-boop]

Yeah, I was testing, and there was another model that refused to locate the model.

What I'm going to try to do next is to route the model info request through the server, since it has the API key.

[seadotline]

Speaking of me, I have no problem downloading the models themselves from the site, but I would find it more convenient to collect all the information about the model through this extension.
So, if it is possible, as an option, (For those who already have a large library of checkpoints, loras, embeddings, etc.), it would be great to have a button to scan the library and download all additional information (notes, images, etc.) without downloading the model itself. (The Civitai Helper extension for Automatic1111 has similar functionality). As for me, it would be more convenient than downloading the model again for this purpose.
If this is possible, of course.

[cdb-boop]

The main reason I did not add it before, is that a full scan would be really slow. However, since you asked, I can add both a single and a full scan.

[cdb-boop]

On the original issue, I am able download the model you linked. (The separate problem I was running into was a gguf file was zipped.)

In the model manager node folder, have you tried putting your api key in the server_settings.yaml file, by setting civitai_api_key: <API_KEY>?

[seadotline]

The main reason I did not add it before, is that a full scan would be really slow. However, since you asked, I can add both a single and a full scan.

Thank you! That would be great!

In the model manager node folder, have you tried putting your api key in the server_settings.yaml file, by setting civitai_api_key: <API_KEY>?

For some reason I didn't know anything about the API key before your question. I added my key in the server_settings.yaml and the problem with downloading models is gone. Thank you!

I also found that some models are downloaded with metadata and some are not, is this the way it should be?

[cdb-boop]

It should download as long as you check the "Save Notes." checkbox at the top of the download tab.

In the settings tab, you can set this to on by default with by checking "Save notes by default."

[seadotline]

Got it, thanks!

And if possible, please add a button to open the model page on Civitai.

[cdb-boop]

Sure thing.

[seadotline]

After checking if “Save notes by default.” is enabled in the settings and "Save Notes" on the download page, I downloaded several models, but still not all models have the “Metadata” tab after downloading.

For example, these models:
https://civitai.com/models/277058/epicrealism-xl
https://civitai.com/models/1116/rpg

[cdb-boop]

The metadata tab is hidden if no embedded metadata was detected in the model.

[cdb-boop]

Okay, I've pushed some changes to my dev fork. There are two new buttons: (1) Try to open model url (I added an option in the settings to show the button also on the preview in model search grid) and (2) try to download model info from the web.

Also, I added a python dependency, so you will need to make sure it is installed.

I did not add a bulk scan functionality, as I am still debating if I should really add it, since a server could theoretically ban someone for automatically scraping a bunch of data all at once.

[seadotline]

Hmm, after updating dev Fork, “Model” button in ComfyUI is not showing up.

As far as I understand, Civitai can temporarily block access to the site in case of excessive loading, but it will not ban.

Through Civitai Helper I have scanned my entire library (about 900 Lora,150 checkpoints and 170 embeddings) and have not encountered with any bans or blocking.

[cdb-boop]

Again, did you install the new requirement? Otherwise, I need the command line error to debug it.

[seadotline]

Sorry, I install the requirements and the button appears.

Unfortunately, the button to open the model page on Civitai doesn't work for me. When I press it, it turns to a gray color for a few seconds and then lights up red. There are no errors in the console after that.

But in the console often (sometimes when trying to download models and sometimes for no clear reason) appears another warning:

Error handling request
Traceback (most recent call last):
  File "C:\Users\vorti\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "C:\Users\vorti\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\web_app.py", line 504, in _handle
    resp = await handler(request)
  File "C:\Users\vorti\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\web_middlewares.py", line 117, in impl
    return await handler(request)
  File "F:\Stable Diffusion\ComfyUI\server.py", line 61, in cache_control
    response: web.Response = await handler(request)
  File "F:\Stable Diffusion\ComfyUI\server.py", line 73, in cors_middleware
    response = await handler(request)
  File "F:\Stable Diffusion\ComfyUI\custom_nodes\ComfyUI-Model-Manager\__init__.py", line 429, in get_model_preview
    uri = uri.replace("/" if sep == "\\" else "/", sep)
AttributeError: 'NoneType' object has no attribute 'replace'

I'm not sure what's causing this message.

Button "download model info", downloads only the notes, but does not download the preview image. Is this how it should be?

[cdb-boop]

Strange. I'm not sure why the uri is None. It should never be none. I put in a fallback safety check, but can you update the node and tell me what the console in the web browser says? It should print PREVIEW_NONE_URI: /model-manager/preview/get?uri=no-preview.

Button "download model info", downloads only the notes, but does not download the preview image. Is this how it should be?

The issue with automatically downloading preview images is that there are multiple, and the image list returned by the API may not be in the "correct" order, e.g. the first one may not be the author's preferred preview image (unless it has changed). Furthermore, I've had issues with using the Civitai API in the past failing to get the original full-resolution image.

[cdb-boop]

Okay, I pushed a fix for properly encoding the model path in url text (for example the character +). See if that fixes anything.

[seadotline]

can you update the node and tell me what the console in the web browser says?

I'm sorry, I don't quite sure how I'm supposed to do this. What do you mean by “update the node” and “console in the web browser”?

I have now completely reinstalled dev fork from scratch and so far everything is as it was, but I have figured out how to launch the page.

If you just click on the Civitai open button, nothing happens, but if you click on that button and then immediately click several times (4 times for example) on the info button, then the page will open. In the console [py.exe] does not write any messages, but the error message that I wrote before is also gone.

[cdb-boop]

Very strange... They should be unrelated. Did you restart ComfyUI?

[seadotline]

Many times. Installed the latest version and I reinstalled requirements.txt once again.

This button behaves very unpredictably. Now I tried after the restart to do the same thing I showed in the gif, and it doesn't react at all now. Another time it opened the Civitai page for me after a minute of waiting, but then it stopped doing that.

And the python command line doesn't show any errors.

[cdb-boop]

I am assuming there is something wrong with my logic with handling the operating system file paths or the browser. Can you say what operating system ComfyUI is running on?

Also, I pushed an update printing the bad url for the failed uri is None error. Can you report any examples of the errors in the command line.

[seadotline]

I am using Windows 10 Home x64

Reinstalled dev fork, now the first time click on the button it shows this message:

Invalid uri! Request url: http://127.0.0.1:8188/model-manager/preview/get

[cdb-boop]

This is helpful. It should be http://127.0.0.1:8188/model-manager/preview/get?uri=no-preview

I'll have a look over the code to see why this is happening.

I am using Windows 10 Home x64

I'm using Windows 11, so maybe that's not the issue. Is your browser up-to-date? I am using Chrome.

[seadotline]

I use two browsers, Vivaldi (Chromium based) and Firefox. Both the latest version. I have now checked both and the same message that I wrote earlier comes up.

[cdb-boop]

I'm sorry this is getting dragged out. I am still having trouble reproducing the issue on my end, which is making it difficult to locate the problem. I made a few small changes, but I don't know if they changed anything.

1. Can you provide a more detailed description of the steps to reproduce the issue? This includes the file name and folder path. The more detailed the better.

2. Are there any error messages in the browser developer console? You can read how to access the browser's console here.

3. Is the issue happening with every model or just certain ones?

4. Is trying to open the external link the only thing that is going wrong with the model manager?

5. Are you browsers preventing pop-ups?

[seadotline]

No problem. :)

I recorded a short video (if you have two monitors, it can be stretched to one and a half that would better read the text).

1. I do not see any specific algorithm to reproduce this error, The button to open the site works very differently.

Which folders and files are you interested in? Where are the models located, or the extension?

2. The console can be seen in the video, I hope this information will be useful.
   There was also an error (not in the video):

Uncaught TypeError: Cannot read properties of undefined (reading 'target')
    at content.js:9:514800
(anonymous) @ content.js:9
requestAnimationFrame
t @ content.js:9
n @ content.js:9
Show 3 more frames

I'm not quite sure what caused it.

3. absolutely with any model.

4. Not really. I noticed that when model manager tries to do something (download, open page, etc) the model info button doesn't work (you can see it in the video). It also seemed to me that Comfyui itself stops rendering the image (while the model is downloading), but I'm not sure it will have to be checked again. That's all I've noticed so far.

5. Yes, my browser blocks popups.

The video will be deleted automatically after 60 minutes.
https://tmpfiles.org/12400055/py_nuyqekzh8r.mp4

[cdb-boop]

Okay, from your video, it appears it appears everything is working normally. It must be something really minor.

The reason I asked if popups were blocked and if it was happening with every model, is I noticed popups were blocked by other browsers I tested. I assume when hashing a checkpoint file, which is large and therefore slow, it takes suspiciously too long for the browser. Just to confirm, does opening the external link not work with small LoRAs or text embedding files? Does it work if you allow popups or disable any ad-blocker?

Also, in you're video didn't have the ComfyUI command line at the bottom and it didn't scroll with it, so I couldn't see if there were any ComfyUI server errors. Are there any abnormal errors there?

[seadotline]

I turned off the ad blocker and when I clicked on the button, browser asked me to allow Civitai popups. After that, small files size started to open the page on Civitai faster and large files took longer.

ComfyUI command line at the bottom

Sorry, I didn't quite understand. Is this command line inside the Comfyui window itself, or do you mean at the bottom of the py.exe console window?

While I was dealing with the blocker, several errors appeared in the [py.exe] console:

Invalid uri! Request url: http://127.0.0.1:8188/model-manager/preview/get
C:\Users\vorti\AppData\Local\Programs\Python\Python310\lib\site-packages\PIL\Image.py:981: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
  warnings.warn(
Invalid uri! Request url: http://127.0.0.1:8188/model-manager/preview/get
Invalid uri! Request url: http://127.0.0.1:8188/model-manager/preview/get
Task exception was never retrieved
future: <Task finished name='Task-721' coro=<RequestHandler.start() done, defined at C:\Users\vorti\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\web_protocol.py:462> exception=AssertionError()>
Traceback (most recent call last):
  File "C:\Users\vorti\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\web_protocol.py", line 505, in start
    request = self._request_factory(message, payload, self, writer, handler)
  File "C:\Users\vorti\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\web_app.py", line 446, in _make_request
    return _cls(
  File "C:\Users\vorti\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\web_request.py", line 811, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\vorti\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\web_request.py", line 188, in __init__
    assert transport is not None
AssertionError

[cdb-boop]

Sorry, I didn't quite understand. Is this command line inside the Comfyui window itself, or do you mean at the bottom of the py.exe console window?

Yes, the ComfyUI command line I was referring to was in the py.exe.

I turned off the ad blocker and when I clicked on the button, browser asked me to allow Civitai popups. After that, small files size started to open the page on Civitai faster and large files took longer.

Do the Invalid uri! Request url: http://127.0.0.1:8188/model-manager/preview/get errors appear when it is disabled? And can you still open the links after you reenable the ad blocker?

Invalid uri! Request url: http://127.0.0.1:8188/model-manager/preview/get

For this error, are there any preview images are missing? Are there any obvious patterns, such as, they all end with ".png"?

[seadotline]

Invalid uri! Request url: http://127.0.0.1:8188/model-manager/preview/get

I think I may have figured out what is causing this error. This is the result of the "Imagus" browser extension. It shows a large resolution image in a popup window.
I recorded a small gif to show it.

[cdb-boop]

Nice. I was able to reproduce it with Imagus. I'll see if I can fix the problem and then I'll get working on your request on batch scan/download notes. For batch scanning images, we'll have to see, since I've had problems getting full-sized images using the Civiatai image API. Again, not to mention this is more likely than notes to get flagged by the server.

BTW, I pushed a change to the dev fork where, if opening a link in a new url is blocked, it will instead open a ComfyUI dialog with a clickable link. Not as pretty, but I think that problem is solved.

[seadotline]

BTW, I pushed a change to the dev fork where, if opening a link in a new url is blocked, it will instead open a ComfyUI dialog with a clickable link. Not as pretty, but I think that problem is solved.

I like this solution even better than the automatic window opening. (at least for the moment).

The only problem I see so far is that when clicking on checkpoints with a size of about 6.62 GiB, it takes about one minute to prepare the link.

[cdb-boop]

Yes, I know it's slow. I may end up caching the hashes and/or urls. I have to take a bit of time to implement it correctly because the model files can be moved around which adds an extra wrinkle.

Alternatively, I recall someone suggested saving Civitai model info next to the model, which could short-circuit the problem, but this requires a bit more thought to implement it, because it needs to work with any website and it needs to be friendly with any other applications that may be accessing the models.

[cdb-boop]

I fixed the issue with Imagus. It should work now.

[seadotline]

I'm not an expert in this, but the only thing I can imagine is to save all the technical information about the model including the link to the site into an existing text document with notes. If of course this is even possible.

[seadotline]

I was wondering if it would be possible to make model-manager, when saving information about a model, create one zip file that would store all information, notes and all images of the model, not just previews. Model-manager would simply view this zip file and show all its contents in its interface.
I realize that MM would have to unzip each model into the Temp folder (Windows) when viewing it, but I don't think it would be a large size.
Just my thoughts. I don't know if this is a good idea in terms of execution.

[cdb-boop]

I don't think there's a need to zip the resources. That seems more like an individual's responsibility and again I want to be compatible with other software out there for model management.

(Model weights are generally noisy and not really compressible, but if they were (e.g. empty weights, specialized lossless compression algorithm), that seems outside the scope of this extension. Images are similar, but there are ways to compress them in a lossy way, which is closer but generally outside the scope of this extension. And, the text we are dealing with is small, so that does not need to be compressed.)

But yes, I'm still working on saving the model urls and info.

[cdb-boop]

I pushed an update to download the website metadata and save it beside the model file. All new downloads will save it the first time. It will use the website metadata to find a link before trying to hash the model, which should speed up opening links on subsequent url openings.

Now I can move onto scanning.

[cdb-boop]

I've pushed another update to save .url files beside the model, which should make opening model links quick (assuming the .url file exists).

[seadotline]

I've pushed another update to save .url files beside the model, which should make opening model links quick (assuming the .url file exists).

It works pretty well, but some models create links not to the Civitai overview page, but a link to download the model.
For example, this is how the DreamShaper 8 link looks like.
https://civitai.com/api/download/models/128713

As I understand, it occurs with models downloaded after the last update. For models downloaded before, link works correctly

[cdb-boop]

You're right. In the downloader, I was saving the file download link.

As I understand, it occurs with models downloaded after the last update. For models downloaded before, link works correctly

Correct. I think I fixed it, but you'll have to delete any bad .url files manually. I can't easily assume it should be ignored, because someone could have saved a url file previously and wanted it to be the file download link.

[seadotline]

Thanks! Model links work great now!

[cdb-boop]

Okay, I've added a scan function to scan all model directories and download and save model info, notes and url. Obviously, it will take a longer if there are big files (e.g. checkpoints) to hash and it will not cache failed searches, so avoid running it multiple times. I tested it on my files and fixed any crashes.

[cdb-boop]

Here's my solution for setting model previews individually within the model manager. You can read more here: #25 (comment)

Again, I don't know if you want to do a scan, as that will select whatever. Note that the images that are found in the model info may not be full resolution or compressed (for example, jpeg instead of png).

[seadotline]

Checked it out, it works fine.

I have a few more future requests, if possible.

1. Add information about the base model (sd1.5, sd2, Pony, Flux, etc). Authors don't always specify this in the title or in the notes, but it is always listed in the appropriate window on Civitai. This information can be very important and now you have to follow the link to find out.
   

2. When clicking on a model name in Load Checkpoint (or Load Lora), the Model-Manager window would open instead of the standard list and when double-clicking on an image, the window would close automatically.
   I don't know how hard it is to do this.

[cdb-boop]

Here is the instructions for doing a scan to download model previews. Note that you must download model info first for it to work properly.

[cdb-boop]

1. Add information about the base model (sd1.5, sd2, Pony, Flux, etc). Authors don't always specify this in the title or in the notes, but it is always listed in the appropriate window on Civitai. This information can be very important and now you have to follow the link to find out.

I agree this is very important and I've looked into this problem in the past. However, I'm not aware of an easy solution.

Reading metadata from files in bulk is SLOW and may be wrong. For example, the metadata produced by Ostris's Flux Dev lora trainer says the base model is "sd_1.5". And, what should be done for files where there are no hints about base model, or for model files that are not image diffusion models (unet, clip, LLMs, etc.)?

What I do, and what I recommend doing, is to manually sort your models into folders by base model and keep them strictly organized.

Doing this has the benifit that you can then search just that model file, for example, search only flux dev models with flux/1.0/dev, or exclude sd1.5 models with -/sd/1.5. For example, here's how I sort my LoRAs:

2. When clicking on a model name in Load Checkpoint (or Load Lora), the Model-Manager window would open instead of the standard list and when double-clicking on an image, the window would close automatically.
   I don't know how hard it is to do this.

Maybe I'm not understanding what you're asking for, but I feel this functionality is a bit specialized and I'd prefer to avoid overriding default ComfyUI behaviors.

---

Also, if you have specific feature suggestions/requests, feel free to open a new issue so it is easier for everyone to keep track of. 😉