As to limit (reduce) the work of the python script?

Question

As to limit (reduce) the work of the python script?

Hi! There is a xlm file with a size of ~300 Mb. It is necessary to get some info and lay down next to the script. Wrote this script:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from xml.dom import minidom

xmldoc = minidom.parse('otchet_tovara.xml')

itemlistcod = xmldoc.getElementsByTagName('cod')
itemlistnaimenov = xmldoc.getElementsByTagName('naimenov')

traine_cod = []
traine_naimenov = []
traine_all = []

for s in itemlistcod:
traine_cod.append(s.attributes['cod'].value)

for s in itemlistnaimenov:
traine_naimenov.append(s.attributes['naimenov'].value)

i = 0
f = open('nelikvid.txt', 'w')

while i<len(traine_address): s1="traine_cod[i]" s2="traine_naimenov[i]" s3=s1 + ' - traine_all.append(s3) f.write(s3, '\n') i="i" 1< code></len(traine_address):>

br><br>
It all works fine when file processing is a small volume. 1-2 Mb. But when you try to process a large file, everything gets a STAKE! The script constantly refers to HDD and the system hangs tightly! To check the usage of RAM is not possible(((<br><br>
Think all the crash occurs when you run this:<br><br><pre><code>for s in itemlistcod:
traine_cod.append(s.attributes['cod'].value)</code></pre><br><br>
What would you recommend? How to fix?

Laney68 asked July 9th 19 at 10:11

Related questions

More answers about "As to limit (reduce) the work of the python script?"

3 answers

Kasey.Cruickshank answered on July 9th 19 at 10:15

Try not to work with the DOM, but use SAX parser, such as lxml. The use will be very different, but will be able to handle files of any size in theory. lxml.de/sax.html
In practice, I chew quietly files this size with no problems.

Wilbert.Murray answered on July 9th 19 at 10:17

The fact that you are using a dom parser to parse an xml file that just has leads to his full load into RAM, and if insufficient memory is available the whole thing starts to sopitsa and firmly stops the car.
Your solution: use to parse xml file sax parser. Examples abound on the web)

glennie_Mer answered on · Accepted Answer · 2019-07-09T10:13:58.231Z

glennie_Mer answered on July 9th 19 at 10:13

Solution

1. To upload a file to the RAM
2. Try a different parser (lxml, beautifulsoup)

To upload a file to the RAM. What is it like? - Laney68 commented on July 9th 19 at 10:16

The DOM model and so the entire document loads into the RAM! We need the SAX parser to use. - Kasey.Cruickshank commented on July 9th 19 at 10:19