How a PHP script can consume 20Gb of memory?

Good day!

On the server run the script, the script has nested loops, accesses files on the server, processes them, and populates the database.
Database some signs have several hundred thousand records, but the sample need only do and index fields is instantaneous. Heavy queries.

Files are stored with names MD5, are divided into folders (.../02/a2/02a263dd93c081eefcf7f2b43e0cde43.html) in under one folder, not more than 10 files.

Using htop to look how it EATS memory, once caught him eating 20GB of RAM, then it stopped and decided to deal with it.

using memory_get_usage() ( or memory_get_usage(true)), the script start eating 10 ... 12... megabytes of memory, all good. But at the same time using htop it already calving 2-3 GB.

Understand asking people to explain to me or give an advice (link) to the understanding of this phenomenon.

Who eats the RAM, why and how to fix it.

Thanks in advance! :-)
July 2nd 19 at 17:28
7 answers
July 2nd 19 at 17:30
Solved the problem. Was being parsed the file using phpQuery, it uses an abstract class, an instance I could not create. Deep is not immersed, but decided to replace it. Replaced by simple html dom, ceased to limit your RAM and plus a significant increase in speed.
Perfectly described the decision:
while(true) {
$html = PhpQuery::newDocument($content);
// work-work-work
} - marielle commented on July 2nd 19 at 17:33
July 2nd 19 at 17:32
Put xhprof and see.
July 2nd 19 at 17:34
Using htop to look how it EATS memory, once caught him eating 20GB RAM

What's that exactly? At least htop shows 3 "kinds" of memory: VIRT, RES, SHR. VIRT can be 20GB. But that does not mean the use 20GB of physical memory.

memory_get_usage will show the current memory consumption. Better to look at memory_get_peak_usage(true). And at the end of the script.
July 2nd 19 at 17:36
July 2nd 19 at 17:38
Perhaps You are now logged database queries and memory increases.
July 2nd 19 at 17:40
Depending on how you files read, especially in a nested loop. Line whether, as a whole, if the whole then by what method, look ob_get_level.
Depending on your solution, try to use generators (yield) if possible. In my opinion you have a problem or ways of reading files.
The second bottleneck is the entry in the database. The insert operation is not always cheap, plus if a lot of them, try to make it transactional.
July 2nd 19 at 17:42
Clean not use return variables, especially arrays.
Clean manually as described here

Find more questions by tags PHP* nix-like systems