Better to write a parser that should work on the server clock?

The task such: on the server there should be a script that will send requests to the api and to store the information in the database. The script should work constantly to our database was always up-to-date information. System CentOS, Debian or Ubuntu. Need the ability to either run multiple parsers, or some multi-threading because the data of tens of millions, and eventually hundreds. Also, it is desirable somehow to control: start, stop, and so he restarted in case of emergency situations. Preferred languages: php, java.
What would you recommend?
September 19th 19 at 12:30
2 answers
September 19th 19 at 12:32
There is a cool thing to control demons . Uses quite elementary - write a console application (daemon) on any convenient language (e.g. php), then indicate what you want supervisord to run 10 or 100 or how many instances of this script, it launches them, monitors so they don't fall off (restarts if anything). It is also possible to stop / restart all these processes at once. Yet supervisord is there a web interface where you can conveniently monitor the state of the demons, plus manage. We use it for your demons for several years and are quite happy.
Looks like this is it, thank you. :) - Esta_Upton commented on September 19th 19 at 12:35
I'd advise to put to him this plugin and in the standard implementation of start/stop/restart is extremely slow. In General, if you need help with supervisord, can knock at Skype. - Austin_Goldner57 commented on September 19th 19 at 12:38
September 19th 19 at 12:34
If it's not roketsan of some sort, then write on everything (Java, Python, Perl, C++), and then see which is better and faster.

Find more questions by tags Demons