How to run a great PHP parser?

Hello!

I have a parser written in PHP using the library Simple HTML DOM. Run the script via CRON. During the execution of the script gives a 502 error.

The script itself such a plan:

$page_list = 30; // the value is parsed

for ($i = $page_list; $i > 0; $i--) {
 foreach ($products as $product) { // 60 products per page
 // Open each item and parsim data
}

 // On average, the parsing of a page (60 items) 30 seconds
}


Prompt how better to launch such a script, so he worked to the end?
June 10th 19 at 14:36
2 answers
June 10th 19 at 14:38
through the console
I run this: /usr/bin/wget -O /dev/null "script.php" using CRON
That limits script execution time no. What is the benefit of running via the console? - Shanny.Feest62 commented on June 10th 19 at 14:41
,
apparently wget has some limitations
then run from cron like this:
/usr/bin/php the path to the folder скрипта/script.php - jerod28 commented on June 10th 19 at 14:44
Thanks, I'll try) - Shanny.Feest62 commented on June 10th 19 at 14:47
way code
$page_list = 30;
for ($i = $page_list; $i > 0; $i--) {

easier to replace
$page_list = $i = 30;
while ($i--) {
- Roderick92 commented on June 10th 19 at 14:50
June 10th 19 at 14:40
// Open each item and parsim data

Then you can optimize. You can download in parallel.
https://github.com/chuyskywalker/rolling-curl
And of course to run the script as a console application. You seems to be using wget or curl some run.
Thank you for reading. Start Yes, using wget is not very good as I understand?) - Shanny.Feest62 commented on June 10th 19 at 14:43
Yes. Is extra overhead to run the wget and the web server. - jerod28 commented on June 10th 19 at 14:46
It is not only unnecessary overhage: puff runs using a web server and therefore is killed by timeout. - Shanny.Feest62 commented on June 10th 19 at 14:49

Find more questions by tags PHP