Civitai - questions and feedback

Post Reply
Nobrumski
Posts: 2
Joined: 19 Mar 2024, 05:38

Civitai - questions and feedback

Post by Nobrumski »

I recently started trying EPF with the Civitai template and wasn't sure if I was missing something. Some images are posted in both Jpeg/JPG and PNG and some are only in Jpeg/JPG. Is there a way to set a preference for PNG and only grab the Jpeg/JPG if no PNG file?

Currently it seems to grab both files. I set it not save duplicates but due to the file type it doesn't see them as a duplicate. With smaller groups of images not a huge deal, but some models can easily have 10k+ images. The PNG images also tend to have the workflow embedded where ComfyUI can read it, but never had any luck getting ComfyUI to read Jpegs.



I was originally using a python tool, and one of the nice features is it creates subfolders with the model name when downloading model images based off the modelversionID. Is there a way to recreate this? I tried sub-folders based off folder structure, but that didn't change anything.
For example model#6424 ends up in the following folders.
model_6424_unknown_model
model_6424_blueberrymix_10
model_6424_chilloutmix.8Rrd-fp16


Also another python tool I tried, which allowed you to select a default sorting method before scraping. This goes back a bit to the first issue where models and posters can have thousands and thousands of images so would prefer to grab the images with the highest stats first. Any chance this could be incorporated? I think when I sorted before pressing continue download it seemed to help with this, but would prefer to automate since I plan to have it checking users and models periodically for new images posted since it was last run.

It looks like the API allows it, but not sure if this is what you are using or doing something else instead.

Thanks.
Maksym
Site Admin
Posts: 2113
Joined: 02 Mar 2009, 17:02

Re: Civitai - questions and feedback

Post by Maksym »

I recently started trying EPF with the Civitai template and wasn't sure if I was missing something. Some images are posted in both Jpeg/JPG and PNG and some are only in Jpeg/JPG. Is there a way to set a preference for PNG and only grab the Jpeg/JPG if no PNG file?

Currently it seems to grab both files. I set it not save duplicates but due to the file type it doesn't see them as a duplicate. With smaller groups of images not a huge deal, but some models can easily have 10k+ images. The PNG images also tend to have the workflow embedded where ComfyUI can read it, but never had any luck getting ComfyUI to read Jpegs.
Doesn't look like it can be done :( You can remove 2 last Custom Parsers from your project and you will stop receiving PNG files. But that's not what you want, do you?
I was originally using a python tool, and one of the nice features is it creates subfolders with the model name when downloading model images based off the modelversionID. Is there a way to recreate this? I tried sub-folders based off folder structure, but that didn't change anything.
For example model#6424 ends up in the following folders.
model_6424_unknown_model
model_6424_blueberrymix_10
model_6424_chilloutmix.8Rrd-fp16

Also another python tool I tried, which allowed you to select a default sorting method before scraping. This goes back a bit to the first issue where models and posters can have thousands and thousands of images so would prefer to grab the images with the highest stats first. Any chance this could be incorporated? I think when I sorted before pressing continue download it seemed to help with this, but would prefer to automate since I plan to have it checking users and models periodically for new images posted since it was last run.
If they allowed to use their API - that could be done (maybe). But they do not allow it anymore (it's so limited, that you won't want to use it). Take a look at the download log of your project to see what URLs Extreme Picture Finder actually uses and which of them can be used for your purposes. Extreme Picture Finder mostly uses URLs for file and sub-folder naming.
Nobrumski
Posts: 2
Joined: 19 Mar 2024, 05:38

Re: Civitai - questions and feedback

Post by Nobrumski »

Maksym wrote: 20 Mar 2024, 15:05 Doesn't look like it can be done :( You can remove 2 last Custom Parsers from your project and you will stop receiving PNG files. But that's not what you want, do you?

If they allowed to use their API - that could be done (maybe). But they do not allow it anymore (it's so limited, that you won't want to use it). Take a look at the download log of your project to see what URLs Extreme Picture Finder actually uses and which of them can be used for your purposes. Extreme Picture Finder mostly uses URLs for file and sub-folder naming.
No I would prefer PNGs and only grab the Jpegs as a backup. The big issue is the workflow data if available is embedded in the PNG for the program to allow you to see how the image was created. I don't believe they have done any updates to allow ComfyUI to read Jpeg data.

For example this is what the log shows so the file name is identical, I don't know if based off that there might be a way to tweak the duplicate setting so it only looks at the actual naming without extension or when it sees duplicate names defaults to the higher image size.

This is from the log.
PNG
jpeg

https://civitai.com/images/8187579 which is sourced from this page https://civitai.com/models/352581. It is one of the 6 preview images. On the model page if you right click save image as, you get the jpeg. If you click on the image and then save it from the actual image page https://civitai.com/images/8187579 you get the PNG. Both unfortunately show a model depth of 3 so I can't just exclude the higher depth.


Yeah I know the API is spotty at best and limited. They just rolled out some updates for it a few days ago so not all sure what was changed. I have been using those two python tools which still work, but was looking for a better way to automate and remember the history like how EPF does which is how I stumbled across EPF. I guess I can try and manually grab the model versionID link instead and create different projects off that.
Post Reply