What file system and with what settings best to use for a very large number of small files?

On the server disk space is expected to 60кк files. 15кк folders.

Structure: 4K folders, each approximately 3750 folders, each with 4 file.

I've found initially 15-20 bytes. Over time will grow 0.6 kilobytes per month for about 30 months disk of 1 terabyte.

Tested at the moment, xfs and ext3, what stood now on the server on two discs.

ext3 - 4KB block size, the drive is much more loaded data.
xfs - block size of 2KB

Speed record:
ext3 5 hours 20 minutes ~18.400.000 created files and folders.
xfs 5 hours 20 minutes ~14.200.000 created files and folders.

The more data recorded, the faster falling speed of the recording. For xfs this time is much worse than ext3, though ext3 is also not very well behaved.

The speed of the first reading of 50K folders using php functions opendir and readdir:
ext3 - ~800 seconds
xfs is ~1200 seconds

On speed reading the same 50K folders, the second and subsequent times:
ext3 - 5seconds
xfs - 47 seconds

But it is only 50K folders. If 15кк folders - when ext3 first complete reading of all folders will be approximately 66 hours. It is necessary for ~12. And I'm afraid that re-reading will not be as fast as in the case of 50K folders, because so much data will not be stored in the cache (or where are they stored?) constantly in full (or will be?). Even if re-reading will happen very quickly, in case reboot the server again need to spend 66 hours in the first reading. So not suitable.

ext4 will work more bright ext3 as I understand it. But hardly will manage to read initially all 15кк folders for ~12 hours using php functions opendir and readdir.

What can something will advise? ext4 is much faster recording/READING than ext3?
What file system to put to test, with what settings? Maybe there are other solutions to the problem of slow reading?
July 2nd 19 at 13:38
2 answers
July 2nd 19 at 13:40
ext4 is certainly better. formatting options -T small for small files
reiserfs is still possible to try

test it on a virtual disk placed in the file (in the same file to exclude the effect of fragmentation)

dd if=/dev/zero of=/home/user/ext4test.img bs=1M count=10000
mkfs.ext4 -T small /home/user/ext4test.img
mkdir /mnt/ext4test
mount /home/user/ext4test.img /mnt/ext4test

test
July 2nd 19 at 13:42
tormented and tested on all available FS except the raiser since he died
had to stop to ext4, especially the option for small files, at least in the base, they shove, but if you add them it's not really suitable
so I recommend to save and make three levels of folders instead of two
the same structure can be stored in the database and not query the folder for other folders , that is to go directly to the file

for formatting and mounting using now something like this
mkfs -t ext4 -m 0 -O dir_index,filetype /dev/vg0/lv0
tune2fs -o journal_data_writeback /dev/vg0/lv0
noatime,nodiratime,noacl,data=writeback,commit=15

the performance is still poor, you just want to sit down and write a database to a small file with the awesome performance

Find more questions by tags ServersHard disksExtended file systemXFS