How to mass check a list of sites for the presence of a page?

Hello. There is a task check list (100,000 URL) for the codes 200 and 404.
The type of web service httpstatus.io only for the desktop and multi-threaded.
April 4th 20 at 12:56
1 answer
April 4th 20 at 12:58
xargs -n1 -P 10 curl -o /dev/null --silent --head --write-out '%{url_effective};%{http_code};%{time_total};%{time_namelookup};%{time_connect};%{size_download};%{speed_download}\n' < the url.lst | tee results.csv
Records like so:
site.com/page.php
0 0.000000 0.000000 0.000000 0 0.000 - Leopold.Les commented on April 4th 20 at 13:01
Oddly
5e26e6f7b30f9579527010.png

The input file with a list of links broken down by lines? - wyatt62 commented on April 4th 20 at 13:04
@wyatt62, Yes.
site1.com/index.php
site2.com/index.php
site3.com/index.php

Tested on Ubuntu. - Leopold.Les commented on April 4th 20 at 13:07

Find more questions by tags Software