PHP script with great run time — how to properly write this?
Welcome all habrovky
There is a script parsing an xml file and then write to the database and downloading images to URLs from this file. The script uses SimpleXML. Immediately say that he wrote it not me, I just now bring it up to the States we need writing necessary things.
The problem is that the size of the file, which we parsim very large — 8000-8500 records. Plus, each entry of 3 to 5 pictures, which he shakes. As a result, the script parses somewhere 6,500 records, and then quietly stops. I tried to run this script on the hosting where it is possible to raise the maximum execution time — it helps but not completely — parse somewhere 7500-7800 entries and stops. Maybe there are some limitations that need to raise?
Please suggest me about the methods of writing such scripts processing ballsie amounts of data. Using cron to run it will not succeed, because this is an extension to one of the TSMs.
I would be grateful for any thoughts and ideas
Edythe.Marks36 answered on October 3rd 19 at 04:28
I designed these scripts in a console command in PHP.
And everything worked according to this principle:
1) Run the command, it writes to the database (in a special plate) that it is running
2) On frontend runs ajax which from time to time checks the sign condition of the team by its ID.
3) the Team failed and recorded in the sign your status "error" for example and the error message, ajax request saw it and reported already to a web interface.
4) Team success — similar to item (3)
carole82 answered on October 3rd 19 at 04:30
and runs as? In the automatic mode or the person runs?
If the latter — AJAX + use function register_shutdown_function () that would catch the moment of failure, to return back a flag that the import is not completed + the number of the last record. Repeat requests automatically, until the task is completed.
samanta_Ho answered on October 3rd 19 at 04:32
I certainly understand that to give such advice when the software has been written is not the best idea, but still advise You to look at the possibility of raspredeleniya process, for example with gearman, it will increase the performance of Your script.
warren_Ankunding answered on October 3rd 19 at 04:34
if the script is run from browser (Ajax, or just yanking links) do not forget to put
Lilla99 answered on October 3rd 19 at 04:36
The SAX parser will work much more efficiently to memory, and most likely a lot faster.
ole_Beier answered on October 3rd 19 at 04:38
You can write a script so it was being parsed for example 100 records and then retain the current position and restarted, started the parsing with stored positions.
Thus solve the problem with the runtime and all the rest
carleton answered on October 3rd 19 at 04:40
1. Written script parcasi XML forms HTML a sign with the url of the pictures.
3. Written a simple html page in two frames: the first frame script of claim 1, the second script of claim 2 with the starting values. This page opens in the browser, then you can go to sleep. The computer preferably keep on a reliable channel in the Internet and UPS to relationship is not interrupted.
agustina_Kuvalis37 answered on October 3rd 19 at 04:42
I have php scripts running in the background via cron sometimes for week's work(working with third party API + pumping images + entry in the database).
If does not help look in the web server logs, perhaps you have the DB fall off due to too long a session. The error will be about the same
MySQL server has gone away
Michaela.Brown answered on October 3rd 19 at 04:44
To download pictures of the same script is entirely optional. I usually just save the url to a file which is then given to wget-y (- i flag). It also happens that external rail (CMS) saves files in its own structure. Therefore, for this case, you can wget-th pictures of the pre-pump in the local filesystem. That would be in the FS address of the picture was the same as the URL you should use -x opinio Here is a typical example of loading:
wget -x-b --user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:18.104.22.168) Gecko/20070725 Firefox/22.214.171.124 rated" --referer="http://examplke.com/" -i img_url_list.txt
Shakes very fast + easy parallelise if you split the file into pieces and run from different servers. I have two servers 200 000 pictures a total volume of ~8GB to boot for about an hour.