How to work with the sample from Mongodb in order to avoid unnecessary requests to the database?

Question

How to work with the sample from Mongodb in order to avoid unnecessary requests to the database?

Good morning.

Situation: I get Mongodb from a number of recordings to fit my parameters, example:

tasks_query = task_base.find({
 "$and": [{
 "order_id": x
 }, {
 "site": y
 }, {
 "category": z
}]
})

Then I need a few thousand times tsikljami go through this data in a loop, making the selection of the Cursor by a certain parameter, figuratively:

for task in task_query:
 if task['order_id'] = 234:
 return task['category']

The bottom line: I don't want to make a few thousand calls to the database, and I want to get one large sample into memory and then do iterations on it. But:

If I make a sample from a database into a list of dictionaries, I need to test the whole sample for cycles, to find that very long and resource-intensive
When turning to the dictionary will be +- the same for the indices in different samples are different.

Question: if I just save the resulting Mongo Cursor into a variable, then will I be able to draw samples from it using find() as usual from the base, without having to apply each time to the database?

If there is some another solution (e.g. another structure data supporting index) or have I missed something important, I would be grateful for any advice :)

Jarrett_Bailey asked June 8th 19 at 16:32

Related questions

More answers about "How to work with the sample from Mongodb in order to avoid unnecessary requests to the database?"

2 answers

Bianka0 answered on June 8th 19 at 16:36

Then I need a few thousand times tsikljami go...

It makes no sense, because the result will be only one order_id.

When turning to the dictionary will be +- the same

One dictionary - one index if you need multiple indexes, do a few dictionaries. This is the fastest O(1), if all memory intermeddle. To do this, and monga person is not needed.

But I don't think you have this critical speed, so it is best steamed and used to Mongu as is, it is fast enough.

Used Mongo with compound_index of the four parameters, an iteration dropped from 2.4 seconds to 0.5 seconds, the amount of saved about an hour of live time for each sample/generation, so that Mongo really helped, it happened faster than generation 4 dictionaries :) - Jarrett_Bailey commented on June 8th 19 at 16:39

Shakira.Cruickshank answered on · Accepted Answer · 2019-06-08T16:34:55.387Z

Shakira.Cruickshank answered on June 8th 19 at 16:34

Solution

Mongo is incredibly quick read. Let it is searching and sorting, not python; for this purpose it is intended. Don't be afraid to ship it requests.
In the end, make a very simple test. Take your time and describe the necessary functionality in both versions: monga and Python. Measure the operating time. And everything will fall into place.

Understand reasonable, thanks :) - Jarrett_Bailey commented on June 8th 19 at 16:37

And one more question: what is better - to pass to the function the collection, or set a connection to the database in each function separately? :) - Jarrett_Bailey commented on June 8th 19 at 16:40

don't see that much difference. It is important to do so, so you do not have to refactor the whole code, if you are going to remake something. If the function needs to process the data, need not know anything about their origin. She reported she worked, and returned the result. Then it will be easy to alter the source in the future. - Shakira.Cruickshank commented on June 8th 19 at 16:43

understood, thanks again :) - Jarrett_Bailey commented on June 8th 19 at 16:46