Page 1 of 1

Speed question

Posted: 13 Mar 2013, 03:49
by temmet
Before I begin I would like to say thanks this is a great product.
After running for a day or so is a significant drop in speed expected. I notice that whenever I run a project it will run something like 1000 images an hour for the first few hours then drop over time until it sticks at around 200 or so. I would not be too surprised if this was the site itself just throwing a cap at me but could it be something else. And if it could be something else what should I try with?
Thanks.

Re: Speed question

Posted: 13 Mar 2013, 11:51
by Maksym
Sometimes using more "Excluded URL" filters help. Often the program gets stuck in the addresses that do not lead to images and it's a good idea to exclude those addresses. Also sometimes you should exclude certain URL parameters as they do not offer new content (for example the same image list is sorted in different ways using URL parameters "?sort=size", "?sort=date" and so on).

Anyway, the program will slow down after a certain amount of processed pages because the download log is getting bigger and bigger with every new page and the program needs more and more time to search the log for every page address it adds to the download queue (to avoid duplicates). So, excluding maximum number of unnecessary pages using Excluded URLs can dramatically increase the general download speed.

Re: Speed question

Posted: 14 Mar 2013, 04:02
by temmet
Is there perhaps a way to manually clean that log up a bit if one of the following two cases is true
1) You have added new URL parameters thus making previously hit sites useless and only taking up lines in the log.
or
2) You are under a site which you are certain cannot hit your already downloaded images URL again because of layout meaning that it must hit another logged URL from your log in order to again reach it.
Thanks.

Re: Speed question

Posted: 14 Mar 2013, 11:30
by Maksym
"Clean log" functionality is not implemented in the program. But download log is (almost) a plain text file. Its location is shown in the project properties window, the [ Project file ] section. So you can edit it in any text editor.