Parser discount goods in dns-shop?

I think your question I'm in the right place..
Generally I think a little bit to put the old page: https://www.dns-shop.ru/catalog/markdown/
But it so happened that initially there are only 20 goods, and which would load other items you need to ease off the request for the page
https://www.dns-shop.ru/catalogMarkdown/category/update/
and the answer is the following n products. Here only I as did not realize what kind of request we need to establish that the page gave the answer? Can anyone tell? Thank you.
PS the city is not important, he with the help of the cook is transmitted..
June 3rd 19 at 19:19
2 answers
June 3rd 19 at 19:21
Solution
In what language you write the parser? If you need a stability result and no further problems when changes in the fields and defenses, you can use Selenium. Load the page, take the data, press the next button and so on schedule.
Otherwise use the Board and implement the reading and transfer of the CSRF token.
I write on a pearl, the daemon start script. Here I'm just not caught up in what their secret data transfer to get the answer. Not good at parsers. Will try, thanks. - Marietta commented on June 3rd 19 at 19:24
, looked at the site studied. It's simple. Loading of goods is the link:
https://www.dns-shop.ru/catalogMarkdown/category/update/?offset=40

where offset is the number of skipped products (shift).
Picked up the minimum number of headers required for a query:
GET <a href="https://www.dns-shop.ru/catalogMarkdown/category/update/?offset=40">https://www.dns-shop.ru/catalogMarkdown/category/update/?offset=40</a> HTTP/1.1
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 YaBrowser/18.6.1.770 Yowser/2.5 Safari/537.36
Host: www.dns-shop.ru

As you can see, should contain the User-Arent and title X-Requested-With to a fixed value. Form request, delivered to the offset values starting from 0 (first page) and then at +20 (so done on the site itself). Parse JSON and parasite the value of the html.
Pulled from the web sample code in Perl and remade, something like this:
use LWP::UserAgent;

my $ua = LWP::UserAgent->new;

my $server_endpoint = "https://www.dns-shop.ru/catalogMarkdown/category/update/?offset=40";

# set custom HTTP request header fields
my $req = HTTP::Request->new(GET => $server_endpoint);
$req->header('X-Requested-With' => 'XMLHttpRequest');

my $resp = $ua->request($req);
if ($resp->is_success) {
 my $message = $resp->decoded_content;
 print "Received reply: $message\n";
}
else {
 print "HTTP GET error code: ", $resp->code, "\n";
 print "HTTP GET error message: ", $resp->message, "\n";
}
- deontae commented on June 3rd 19 at 19:27
, Basebase) Count all the work for me))
PS the path in the telegrams you want to do with fast notifications. Sometimes there are interesting slips. - Marietta commented on June 3rd 19 at 19:30
June 3rd 19 at 19:23
https://www.dns-shop.ru/catalogMarkdown/category/u...
using such titles as X-CSRF-Token & X-Requested-With: XMLHttpRequest

Find more questions by tags JSONParsing