How it works search program with large amounts of data?

Are the bots when crawling web documents, there is no scheme of building the search index, like the categorization of "animals", "computers", "money" it.d. so entering a query of "animals" we were looking for in the category "animals" without lifting the entire search database, I walk around in every page. Otherwise, this is how you will have to do iterations to meet the user.
April 4th 20 at 00:32
4 answers
April 4th 20 at 00:34
Solution
In the first approximation https://yandex.ru/company/technologies/matrixnet/

Well, in General the whole section
https://yandex.ru/company/technologies

But no reality you will not find anywhere else.
April 4th 20 at 00:36
Solution
The analogue of the index on the "trees."
Each segmentation is "the tree" links. Their intersection gives the right branch to search only within this narrow category.
April 4th 20 at 00:38
Search engine applies at least the reverse index technology. I advise you to read
April 4th 20 at 00:40
Naive questions for those who have not encountered implementations. Okay, let's start to realize: oops, but how to take all of these categories and how to fill them, okay, somehow it all done, oops, and then there's the homonyms surfaced, okay, somehow it all did. And so, at each iteration of implementation, there is additional complexity.

Find more questions by tags Search engines