Extreme Internet Software forum

I saw an older question in 2016 on how to scrape 500px.com, but as of 2017, the site layout is a bit different now. I'd like to know how to scrape it for high res jpgs at this point.

• Assuming I have a list of starting address -- https://500px.com/alyabev, https://500px.com/sergeybondarev, etc.

• I'd like to auto create a subdirectory for each photographer, e.g. /alyabev/, /sergeybondarev/, etc.

• Each starting page is a really a TGP, so https://500px.com/sergeybondarev points to pages like https://500px.com/photo/234377907/when- ... id=1390899

• In this case, the high res jpg should be named "234377907.jpg" from the above page. So the resulting file will be in "/sergeybondarev/234377907.jpg".

• The website uses AJAX, so using a browser, only the first 50 photos are loaded until you scroll toward the bottom. Of course, I am hoping I can scrape all photos on a page rather than the first 50 only.

Can someone help with the projects setup? Thanks!

OK, looks like I was able to create a template, but it's not 100% automatic, so you can use only 1 starting URL per project. Please read the template instructions carefully:

500px.com - author template

Also you will need the most recent version of the program for this template to work:

http://www.webimagedownloader.com/downl ... rSetup.exe

Very nice!

Just a minor issue, why are some files downloaded multiple times, with 001, 002 endings?

Such as https://500px.com/frbr.

You can modify the template and select "Do not download and save new file if file size is the same" option in the [Save -> Conflicts] section. It should help.