BDD
Matériel
2 serveurs OVH Advance-4 Gamme 2026 (1 Master / 1 Slave) :
CPU : AMD EPYC 4585PX - 16c/32t - 4.3GHz/5.7GHz
RAM : 128GB DDR5 5600MHz
Disques : 4×7.68 To SSD NVMe Soft RAID
1 serveur OVH Advance-STOR Gamme 2026 (Backup) :
CPU : AMD EPYC 4345P - 8c/16t - 3.8 GHz/5.5 GHz
RAM : 128GB DDR5 5600MHz
Disques : 2×960 Go SSD NVMe Soft RAID + 4×24 To HDD SAS Soft RAID
Installer les serveurs MASTER/SLAVE
OS
Partitionnement
L’installeur OVH utilise 2 disques sur 4 au hasard (ici nvme2 et nvme3) :
# fdisk -l /dev/nvme*n1
Disk /dev/nvme0n1: 6.99 TiB, 7681501126656 bytes, 15002931888 sectors
Disk model: SAMSUNG MZQL27T6HBLA-00A07
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: F57BB925-0B46-4398-826A-1EBD837A902D
Device Start End Sectors Size Type
Disk /dev/nvme1n1: 6.99 TiB, 7681501126656 bytes, 15002931888 sectors
Disk model: SAMSUNG MZQL27T6HBLA-00A07
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: F57BB925-0B46-4398-826A-1EBD837A902D
Device Start End Sectors Size Type
Disk /dev/nvme2n1: 6.99 TiB, 7681501126656 bytes, 15002931888 sectors
Disk model: SAMSUNG MZQL27T6HBLA-00A07
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: F57BB925-0B46-4398-826A-1EBD837A902D
Device Start End Sectors Size Type
/dev/nvme2n1p1 2048 1048575 1046528 511M EFI System
/dev/nvme2n1p2 1048576 3145727 2097152 1G Linux RAID
/dev/nvme2n1p3 3145728 15002927103 14999781376 7T Linux filesystem
Disk /dev/nvme3n1: 6.99 TiB, 7681501126656 bytes, 15002931888 sectors
Disk model: SAMSUNG MZQL27T6HBLA-00A07
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: 1E48E8CA-E071-4980-A603-1BAB6CA2827B
Device Start End Sectors Size Type
/dev/nvme3n1p1 2048 1048575 1046528 511M EFI System
/dev/nvme3n1p2 1048576 3145727 2097152 1G Linux RAID
/dev/nvme3n1p3 3145728 15002927103 14999781376 7T Linux filesystem
Copier le partitionnement sur les disques vierges :
sgdisk --replicate=/dev/nvme0n1 /dev/nvme2n1
sgdisk --replicate=/dev/nvme1n1 /dev/nvme2n1
Partition EFI (cf. https://wiki.evolix.org/HowtoRAIDLogiciel#partition-efi)
Dans le cas d’un serveur qui démarre en EFI, il peut être utile d’avoir au moins deux entrées entrées pour pouvoir démarrer même lorsque le disque qui héberge la partition EFI est hors service. Il n’est pas possible de mettre la partition EFI dans du RAID logiciel, donc il faut dupliquer la partition à la main.
cp /dev/nvme2n1p1 /dev/nvme0n1p1
cp /dev/nvme2n1p1 /dev/nvme1n1p1
apt install efibootmgr
efibootmgr -c -g -d /dev/nvme0n1 -p 1 -L debian -l '\EFI\debian\grubx64.efi'
efibootmgr -c -g -d /dev/nvme1n1 -p 1 -L debian -l '\EFI\debian\grubx64.efi'
Partition /boot
L’installeur OVH a configuré un RAID 1 soft avec les deux premiers disques. On va ajouter les deux autres en plus :
root@ns3264579:/home/debian# cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme3n1p2[0] nvme2n1p2[1]
1046528 blocks super 1.2 [2/2] [UU]
unused devices: <none>
root@ns3264579:/home/debian# mdadm /dev/md2 --add /dev/nvme0n1p2
mdadm: added /dev/nvme0n1p2
root@ns3264579:/home/debian# mdadm /dev/md2 --add /dev/nvme1n1p2
mdadm: added /dev/nvme1n1p2
root@ns3264579:/home/debian# cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme1n1p2[3](S) nvme0n1p2[2](S) nvme3n1p2[0] nvme2n1p2[1]
1046528 blocks super 1.2 [2/2] [UU]
unused devices: <none>
ZFS RAID10
L’installeur OVH a normalement créé un pool ZFS mirror :
root@ns3264579:/home/debian# zpool status
pool: zp0
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zp0 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme2n1p3 ONLINE 0 0 0
nvme3n1p3 ONLINE 0 0 0
Il faut maintenant ajouter un autre vdev mirroir pour avoir un équivalent RAID10 :
root@ns3264579:/home/debian# zpool add zp0 mirror /dev/nvme0n1p3 /dev/nvme1n1p3
root@ns3264579:/home/debian# zpool status
pool: zp0
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zp0 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme2n1p3 ONLINE 0 0 0
nvme3n1p3 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
nvme0n1p3 ONLINE 0 0 0
nvme1n1p3 ONLINE 0 0 0
Enfin il faut mettre à jour les quotas :
root@ns3264579:/home/debian# zfs get quota,refquota zp0/zd0
NAME PROPERTY VALUE SOURCE
zp0/zd0 quota 6.98T local
zp0/zd0 refquota none default
root@ns3264579:/home/debian# zfs set quota=none zp0/zd0
root@ns3264579:/home/debian# zfs get quota,refquota zp0/zd0
NAME PROPERTY VALUE SOURCE
zp0/zd0 quota none local
zp0/zd0 refquota none default
root@ns3264579:/home/debian# df -h /
Filesystem Size Used Avail Use% Mounted on
zp0/zd0 14T 1.5G 14T 1% /
Optimisation ZFS/MariaDB
Afin de tirer parti des performances de ZFS, il faut séparer les LOGS et les DATAS. Pour créer les datasets :
zfs create -p -o recordsize=16K -o compression=lz4 -o atime=off -o primarycache=metadata -o logbias=throughput -o mountpoint=/mariadb-data zp0/mariadb/data
zfs create -o compression=off -o sync=always -o primarycache=metadata -o mountpoint=/mariadb-logs zp0/mariadb/logs
chown mysql:mysql /mariadb-data
chown mysql:mysql /mariadb-logs
Installer le serveur de BACKUP
OS
Partitionnement
Note : remplacer - par _ dans le nom des pools ZFS sinon l’installeur OVH plante.
Optimisation disque SAS
apt install openseachest
openSeaChest_Firmware -d all --modelMatch ST24000NM005H --downloadFW SummitExosSAS-SED-512E-ET07.LOD
openSeaChest_Configure -d all --modelMatch ST24000NM005H --writeCache info
for i in $(seq 0 3); do echo "write through" > /sys/class/scsi_disk/2\:0\:$i\:0/cache_type; done
ZFS
L’installeur OVH a normalement créé un pool ZFS mirror et un pool ZFS stripping avec les deux disques NVMe :
root@ov111:/home/debian# zpool status
pool: raid0_nvme
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
raid0_nvme ONLINE 0 0 0
nvme1n1p4 ONLINE 0 0 0
nvme0n1p4 ONLINE 0 0 0
errors: No known data errors
pool: raid1_nvme
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
raid1_nvme ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme1n1p3 ONLINE 0 0 0
nvme0n1p3 ONLINE 0 0 0
On e les réglages par défaut de l’installeur OVH sur le pool raid0_nvme & raid1_nvme :
zfs set mountpoint=none raid0_nvme/zd1 && rm /toremove
zfs destroy raid0_nvme/zd1
zfs set compression=lz4 raid1_nvme/zd0
root@ov111:/home/debian# zfs list
NAME USED AVAIL REFER MOUNTPOINT
raid0_nvme 732K 566G 96K none
raid1_nvme 1.39G 576G 96K none
raid1_nvme/zd0 1.39G 576G 1.39G /
Il faut maintenant créer un autre pool (équivalent RAID10) avec les disques SAS :
Si le serveur a déjà été installé, il faut vider les disques SAS : wipefs -a /dev/sda /dev/sdb /dev/sdc /dev/sdd
zpool create raid10_hdd mirror /dev/sda /dev/sdb
zpool add raid10_hdd mirror /dev/sdc /dev/sdd
zfs set mountpoint=none raid10_hdd
root@ov111:/home/debian# zfs list
NAME USED AVAIL REFER MOUNTPOINT
raid0_nvme 732K 566G 96K none
raid10_hdd 576K 43.5T 96K none
raid1_nvme 1.39G 576G 96K none
raid1_nvme/zd0 1.39G 576G 1.39G /
root@ov111:/home/debian# zpool status raid10_hdd
pool: raid10_hdd
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
raid10_hdd ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
On ajoute un volume special sur le pool raid1_nvme pour les meta ZFS du pool raid10_hdd :
zfs create -V 300G -o compression=lz4 raid1_nvme/meta_for_raid10_hdd
zpool add -f raid10_hdd special /dev/zvol/raid1_nvme/meta_for_raid10_hdd
root@ov111:/home/debian# zpool status raid10_hdd
pool: raid10_hdd
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
raid10_hdd ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
special
zvol/raid1_nvme/meta_for_raid10_hdd ONLINE 0 0 0
On ajoute un volume special sur le pool raid0_nvme pour le cache ZFS du pool raid10_hdd :
zfs create -V 150G -o compression=lz4 raid0_nvme/l2arc_for_raid10_hdd
zpool add raid10_hdd cache /dev/zvol/raid0_nvme/l2arc_for_raid10_hdd
root@ov111:/home/debian# zpool status raid10_hdd
pool: raid10_hdd
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
raid10_hdd ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
special
zvol/raid1_nvme/meta_for_raid10_hdd ONLINE 0 0 0
cache
zvol/raid0_nvme/l2arc_for_raid10_hdd ONLINE 0 0 0
On créé la partition de cache RESTIC :
zfs create -o atime=off -o sync=disabled -o compression=lz4 -o xattr=sa -o mountpoint=/data raid0_nvme/restic_cache
Optimisation ZFS/MariaDB
Afin de tirer parti des performances de ZFS, il faut séparer les LOGS et les DATAS. Pour créer les datasets : Attention : le dataset des logs est créé sur le pool ZFS des disques NVMe
zfs create -p -o recordsize=16K -o compression=lz4 -o atime=off -o primarycache=metadata -o logbias=throughput -o mountpoint=/mariadb-data raid10_hdd/mariadb/data
zfs create -p -o compression=lz4 -o sync=always -o primarycache=metadata -o mountpoint=/mariadb-logs raid1_nvme/mariadb/logs
zfs create -o mountpoint=/mariadb-dumps zp1/mariadb/dumps
zfs create -p -o compression=lz4 -o atime=off -o recordsize=1M -o mountpoint=/mariadb-dumps raid10_hdd/mariadb/dumps
Résultat :
root@ov111:/home/debian# zpool status
pool: raid0_nvme
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
raid0_nvme ONLINE 0 0 0
nvme1n1p4 ONLINE 0 0 0
nvme0n1p4 ONLINE 0 0 0
errors: No known data errors
pool: raid10_hdd
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
raid10_hdd ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
special
zvol/raid1_nvme/meta_for_raid10_hdd ONLINE 0 0 0
cache
zvol/raid0_nvme/l2arc_for_raid10_hdd ONLINE 0 0 0
errors: No known data errors
pool: raid1_nvme
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
raid1_nvme ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme1n1p3 ONLINE 0 0 0
nvme0n1p3 ONLINE 0 0 0
root@ov111:/home/debian# zfs list
NAME USED AVAIL REFER MOUNTPOINT
raid0_nvme 152G 414G 96K none
raid0_nvme/l2arc_for_raid10_hdd 152G 566G 456K -
raid0_nvme/restic_cache 96K 414G 96K /data
raid10_hdd 3.70M 43.5T 96K none
raid10_hdd/mariadb 1.12M 43.5T 384K none
raid10_hdd/mariadb/data 384K 43.5T 384K /mariadb-data
raid10_hdd/mariadb/dumps 384K 43.5T 384K /mariadb-dumps
raid1_nvme 306G 271G 96K none
raid1_nvme/mariadb 192K 271G 96K none
raid1_nvme/mariadb/logs 96K 271G 96K /mariadb-logs
raid1_nvme/meta_for_raid10_hdd 305G 576G 2.34M -
raid1_nvme/zd0 1.59G 271G 1.59G /
Ansible
Si besoin de debug ajouter : “-vvv -e ‘debops__no_log=false’”
ansible-playbook -i inventory/100_hosts playbooks/debops_bootstrap.yml --vault-id @prompt --flush-cache --user debian --limit <server>.pkgdata.net
ansible-playbook -i inventory/100_hosts playbooks/debops_site.yml --vault-id @prompt --flush-cache --limit <server>.pkgdata.net
chown mysql:mysql /mariadb-data /mariadb-logs
mv /mariadb-data/ib_logfile0 /mariadb-logs/ib_logfile0
/etc/init.d/mariadb restart
ansible-playbook -i inventory/100_hosts playbooks/debops_site.yml --vault-id @prompt --flush-cache --limit <server>.pkgdata.net
cd ~ && yadm checkout .
reboot
Troubleshooting
Au reboot le pool ZFS raid0_nvme n’est pas importé automatiquement :
root@ov111:/home/pkgdata # zpool status
pool: raid10_hdd
state: ONLINE
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
scan: resilvered 4K in 00:00:00 with 0 errors on Thu Jun 11 18:41:41 2026
config:
NAME STATE READ WRITE CKSUM
raid10_hdd ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdd ONLINE 0 0 0
sda ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
special
zvol/raid1_nvme/meta_for_raid10_hdd ONLINE 0 0 0
cache
zvol/raid0_nvme/l2arc_for_raid10_hdd UNAVAIL 0 0 0
errors: No known data errors
pool: raid1_nvme
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
raid1_nvme ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme0n1p3 ONLINE 0 0 0
nvme1n1p3 ONLINE 0 0 0
Pour résoudre :
cat >> /etc/crontab <<EOF
@reboot root zpool import raid0_nvme
EOF
Synchro Master -> Backup
ssh ov111.pkgdata.net
sudo /etc/init.d/mariadb stop
ssh ov110.pkgdata.net
mariadb -e "stop slave"
mariadb -e "reset master"
zfs snapshot -r zp0/mariadb@snap
mariadb -e "start slave"
zfs send zp0/mariadb/logs@snap | ssh ov111.bdd zfs recv -F raid1_nvme/mariadb/logs
zfs send zp0/mariadb/data@snap | ssh ov111.bdd zfs recv -F raid10_hdd/mariadb/data
zfs destroy zp0/mariadb/logs@snap
zfs destroy zp0/mariadb/data@snap
zfs destroy zp0/mariadb@snap
ssh ov111.pkgdata.net
/etc/init.d/mariadb start