How to recover a ZFS RAIDZ1?

This :
Proxmox 6.1
RAIDZ1 + ssd cache + ssd log
4 drive-2 TB-WDC_WD2003FYYS ( worked on average more than 5 years)
+ Nvme Samsung_SSD_970_PRO_512GB

Sunday night hanged the server.
Soft reboot didn't help, did the reset
Failed to load.
Connected the monitor and saw that the download could not be read in the sector.
Booted from the stick and ran the program in Victoria drive which was the problem .
Smart stated that there are 1 unstable sector.
Tested and reassigned it ( only 1 sector).
Not having completed the test drive - again tried to boot in Proxmox

Below are attached screenshots.
5e290a0a7718d444519334.jpeg

5e290a21b253d144433168.jpeg

Tried different ways to mount the pool - does not work.
At the moment, ends up creating a full backup of the disk images using ddrescue

That's what it became.
/dev/disk/by-id/ata-WDC_WD2003FYYS-02W0B1_WD-WCAY00923792-part3 disk could zabekapit only 99.9 %
Hundreds of readable sectors.

3 disks have the same TXG ( one of them broken)
1 Disc has its own TXG ( the disc itself is read)

Created disk images.
Mounted these images
losetup /dev/loop1 /opt/ata-WDC_WD2003FYYS-02W0B1_WD-WCAY00923209-part3.bin
losetup /dev/loop2 /opt/ata-WDC_WD2003FYYS-02W0B1_WD-WCAY00925258-part3.bin
losetup /dev/loop5 /opt/ata-WDC_WD2003FYYS-02W0B1_WD-WCAY00923792-part3.bin
losetup /dev/loop0 /opt/ata-WDC_WD2003FYYS-02W0B1_WD-WCAY00923050-part3.bin
losetup /dev/loop3 /opt/nvme-Samsung_SSD_970_PRO_512GB_S463NF0ka29555r-part1.bin
losetup /dev/loop4 /opt/nvme-Samsung_SSD_970_PRO_512GB_S463NF0ka29555r-part2.bin


Tried the PR utility to rollback to the total TXG
https://gist.github.com/jshoward/5685757

Then tried to mount the pool with an indication that TXG
zpool import -fF -XT 922580 -o readonly=on rpool
zpool import -fF -XT 962577 -o readonly=on rpool


The computer "thought" day, but then still couldn't mount the pool

On this pool I essentially need to pull 2-3 of the image and possible backups.

Are there any repair utilities?
This prog can "see" my pool
https://www.ufsexplorer.com/articles/how-to/recove...
But it is paid and not the fact that recover.

Tried this
https://github.com/Stefan311/ZfsSpy
Gives a lot of errors, can not be considered pool

With this script and could not understand to the end. Documentation at a minimum :-(
https://github.com/hiliev/py-zfs-rescue

Fork of the previous project.
Documentation more, but also not fully clear how it works.
Why it starts a local server it is not clear how to run the script.
https://github.com/eiselekd/dumpbin-py-zfs-rescue

Maybe there are some ways to recover data from razvalistoj ZFS RAIDZ1 ?
What can you recommend?
April 4th 20 at 13:26
6 answers
April 4th 20 at 13:28
First, it was necessary to consider - https://docs.oracle.com/cd/E19253-01/819-5461/gbct... is to try to roll back the transaction.

To look at similar problems https://www.ixsystems.com/community/threads/pool-m...

At worst remove the problematic disc and you'd have a Degraded Pool.

We need to understand what is pool metadata and how it is stored. We need to understand what is scrub in ZFS and why is it necessary to use ECC memory. We must understand that the reallocate sector is unlikely to solve the problem since the data is already corrupted.

And then in Lab to go.

And backups to be configured.
April 4th 20 at 13:30
Try to run rescue mode with the image of proxmox
Cannot import pool - Jerod_Champl commented on April 4th 20 at 13:33
April 4th 20 at 13:32
How about pull out the dead drive and try the import without it?
Tried , does not help.
Disconnected 1 disk.
Always the same FAULTED when trying to import pool - Jerod_Champl commented on April 4th 20 at 13:35
@Jerod_Champl, irc ZoL to ask it is necessary and in the newsletter. - dillon.Roo commented on April 4th 20 at 13:38
@dillon.Roo, you can still try -FX, once the images have - dillon.Roo commented on April 4th 20 at 13:41
April 4th 20 at 13:34
And how is it possible for zfs to bring to this state? There scrubbing should work
Schedule 1 time per week.
About 2 weeks ago there was 1 error
Made clear , and then walked manually SRUB and she was gone. - Jerod_Champl commented on April 4th 20 at 13:37
April 4th 20 at 13:36
Try Klennet ZFS Recovery. Certainly not free but the price is quite lifting.
In trial mode will scan your disks and show all that I could find, including the remote.
My first event was recreated over the existing pool, the second pool nemontirani damaged.
Tried to scan this app, but for days nothing happened. Probably because of the bad drive.
Thanks, I'll try again, now I have made images with them should be easier. - Jerod_Champl commented on April 4th 20 at 13:39
April 4th 20 at 13:38
Are there any repair utilities?
This prog can "see" my pool

The data recovery software is better than UFS Explorer, you will find very unlikely.

In trial mode you can use it to study in detail what is on your images (a good thing started with this), and the results to make a decision about buying a key. Limitation of the free version - only retrieval of data.

In some cases, hands can do more than a automatic, but taking into account specifics of ZFS, it is not easy, as a result, is often expensive.

We are remote data recovery (online), if you really run into a wall, you can see, however, in the case of ZFS in advance to promise something unreasonable.

Find more questions by tags ZFSRAID