Where to store infinity records (111 * 10^29)?

Hello. You need to keep records in the form of key => value. The number of records 111 * 10^29 (according to my algorithm). The key - number is either a long string up to 35 characters value is long string up to 60 characters. Question: where can I store such a huge pile of data? Time to search in this database is 20-25 seconds. Understand that volume of data is huge, so you can store the database in RAM (how much it eats memory on different languages?), is it possible?

Thanks in advance.
July 12th 19 at 17:25
7 answers
July 12th 19 at 17:27
This is what are you going to store?
95*111*1029 bytes ≈ 1034 bytes ≈ 1022 terabytes
So you buy a thousand million of millions of millions of hard drives with 10 Tb and get your store.
And 95 from? - Nicolette.Bode commented on July 12th 19 at 17:30
: 35+60 - Mariam_Renner79 commented on July 12th 19 at 17:33
: 95 bytes = 35 characters + 60 characters - augusta commented on July 12th 19 at 17:36
Okay, as if to clarify? The possibility of collisions allowed when using SHA-224 in value? How many values can be passed? - Nicolette.Bode commented on July 12th 19 at 17:39
Interestingly, for such data? It's to fill a need 39861173208808608 years, with an AVERAGE write speed of 800MB/s
((95*111*10**29)/(800*1024*1024))/(60*60*24*365) - augusta commented on July 12th 19 at 17:42
The possibility of collision in SHA is always there, it's a hash. - Mariam_Renner79 commented on July 12th 19 at 17:45
: If I have a database of hashes, they can earn money exponentially) - Nicolette.Bode commented on July 12th 19 at 17:48
Hmm, how much time it takes something to generate a database of numbers with a size of 99 * 10^29 columns, for example, using a powerful dedicated server and amazon dynamodb? - Nicolette.Bode commented on July 12th 19 at 17:51
: Well, well! Success! But what will you do with 2048 caches? And with 4096? It is atoms in the observable universe is not enough to store all values of the caches, I hope You know... - augusta commented on July 12th 19 at 17:54
You "invented" rainbow table, and decided to implement it? Do I understand correctly? - destany34 commented on July 12th 19 at 17:57
: I "invented"? And what's rainbow table?) - Nicolette.Bode commented on July 12th 19 at 18:00
exactly. Read about the rainbow tablicy, it looks like you invented just for her. - destany34 commented on July 12th 19 at 18:03
July 12th 19 at 17:29
On the infinite drive, of course. And how are you this garbage you wrote?
Why me? The developers of one site thought to put the winning number in the hash, only here it is a fractional number with 30 digits after the comma :D - Nicolette.Bode commented on July 12th 19 at 17:32
: the winning number will be matched with the infinity :))) - Mariam_Renner79 commented on July 12th 19 at 17:35
July 12th 19 at 17:31
100 000 000 000 000 000 000 000 000 000 000 000 bytes...
on the flash drive will not fit (
July 12th 19 at 17:33
To store such a large amount of data you can continue there, where they generate - in the algorithm.
PS. What question - such answer.
July 12th 19 at 17:35
Accurate information on price not found. How much will it cost me for such volume of data? - Nicolette.Bode commented on July 12th 19 at 17:38
: https://aws.amazon.com/ru/dynamodb/pricing/ - Mariam_Renner79 commented on July 12th 19 at 17:41
July 12th 19 at 17:37
Let me guess, you decided to find out what is the next number in the double.
Success xD
Phaha, I'm not alone here?))) - Nicolette.Bode commented on July 12th 19 at 17:40
: are you single, because people versed in the subject know that it is impossible at this level of technology. - Mariam_Renner79 commented on July 12th 19 at 17:43
: what is a double? - augusta commented on July 12th 19 at 17:46
: join. What is a double? - Nicolette.Bode commented on July 12th 19 at 17:49
: yo, well I won't sleep all rainbow tables in mind to build, just in case I will... - augusta commented on July 12th 19 at 17:52
: Wow, a person with an infinite mind in the thread. =) - Mariam_Renner79 commented on July 12th 19 at 17:55
: Infinity - no limit! ) But rather a number ) - Nicolette.Bode commented on July 12th 19 at 17:58
July 12th 19 at 17:39
For a start, check again your algorithm. Most likely, you got a good bunch of takes, if not 100%, then some pieces will be exactly repeated. I do not believe that 100% will be some very unique.

The first thing you do is total pieces of their information stand in a separate field. Such a structure as a tree you know? Here is the General piece will has kept you on top of the tree. Further, each node keeps links to nodes with some other unique data, etc. In the principle of nested levels you can have are endless.

To the question where to store. Something better hard drives for this, not yet. In your case it will be wiser to use a hybrid storage SATA + SSD + RAM. The data to which the treatment is most often lie in Redis (i.e. RAM), just often used an SSD, something rarely needed - on SATA. The algorithm of frequency count himself write, by defining for what is often, not very rare.

Provider can provide this on digitalOcean there are tariffs with hybrid screws SATA + SSD, look at him. We also recommend you to look in the direction of docker, in your case, I think, will need 10+ cars for storage, and this thing will allow you to manage their configuration easier.

About the time to retrieve, search etc. Google on "storage trees", "tree search", etc. get away from complete graphs, try to go even cycles even say more, don'T DO a complete graph or a cycle in the graph on this volume, you shoot yourself in the foot just.
Thank you for such a complete answer, but I counted and I have 99 * 10^29 combinations. This is a number from 0.01 * 10^-28 to 1. Even if we take as a basis the fact that is not skipped will need to find a combination (hope that will carry on a certain number at the beginning) then the maximum that I can squeeze 10 zeros. That is 20!! you have to guess and then hope that I'm lucky.

In fact the site simply hashes the winning number and shows the type of fair play. If declassify the hash can be won 24/7 and get the hundred thousand bucks a day) - Nicolette.Bode commented on July 12th 19 at 17:42
: A hash algorithm known? Ready rainbow tables were not looking for him? - Mariam_Renner79 commented on July 12th 19 at 17:45
: Yes, the usual SHA-224, but the table is not your regular, because it is the table of numbers of the form 0.01, and then there's 28 zeros, and so right up to the unit hack is stupidly impossible. - augusta commented on July 12th 19 at 17:48
: Roger that. What about Brutus? will be long, but the generation of the table even more (neobhodimo to go through the whole range of values, not to win + overhead storage). Not been evaluated, how long your algorithm will take to generate (111 * 10^29) of the records? - Nicolette.Bode commented on July 12th 19 at 17:51
: First, I have a limit of 35 seconds. Second, were evaluated. Came 3.16887646 × 10^14 years, and that at a speed of 10 million!! hashes per second. That is the desired speed in ~10^24 hashes per second, which would fit in 35 seconds) - augusta commented on July 12th 19 at 17:54
I think you want the impossible. Even a search on (111 * 10^29) of the records you in 35 seconds won't be enough, IMHO. - Mariam_Renner79 commented on July 12th 19 at 17:57

Find more questions by tags NoSQLSQL