Speed question

Post Reply
temmet
Posts: 2
Joined: 13 Mar 2013, 03:41

Speed question

Post by temmet » 13 Mar 2013, 03:49

Before I begin I would like to say thanks this is a great product.
After running for a day or so is a significant drop in speed expected. I notice that whenever I run a project it will run something like 1000 images an hour for the first few hours then drop over time until it sticks at around 200 or so. I would not be too surprised if this was the site itself just throwing a cap at me but could it be something else. And if it could be something else what should I try with?
Thanks.

Maksym
Site Admin
Posts: 2077
Joined: 02 Mar 2009, 17:02

Re: Speed question

Post by Maksym » 13 Mar 2013, 11:51

Sometimes using more "Excluded URL" filters help. Often the program gets stuck in the addresses that do not lead to images and it's a good idea to exclude those addresses. Also sometimes you should exclude certain URL parameters as they do not offer new content (for example the same image list is sorted in different ways using URL parameters "?sort=size", "?sort=date" and so on).

Anyway, the program will slow down after a certain amount of processed pages because the download log is getting bigger and bigger with every new page and the program needs more and more time to search the log for every page address it adds to the download queue (to avoid duplicates). So, excluding maximum number of unnecessary pages using Excluded URLs can dramatically increase the general download speed.

temmet
Posts: 2
Joined: 13 Mar 2013, 03:41

Re: Speed question

Post by temmet » 14 Mar 2013, 04:02

Is there perhaps a way to manually clean that log up a bit if one of the following two cases is true
1) You have added new URL parameters thus making previously hit sites useless and only taking up lines in the log.
or
2) You are under a site which you are certain cannot hit your already downloaded images URL again because of layout meaning that it must hit another logged URL from your log in order to again reach it.
Thanks.

Maksym
Site Admin
Posts: 2077
Joined: 02 Mar 2009, 17:02

Re: Speed question

Post by Maksym » 14 Mar 2013, 11:30

"Clean log" functionality is not implemented in the program. But download log is (almost) a plain text file. Its location is shown in the project properties window, the [ Project file ] section. So you can edit it in any text editor.

Post Reply