I/O problems on OVH Proxmox LVM2 snapshots

I've recently gave a shot and moved my overloaded server to new one, which uses Proxmox virtualization. I've set up all VMs and put nginx reverse proxy on the top. All things were going nicely, even setting up my very first own firewall and allow some traffic between VM and Internet.

Then I thought I should make backups at least weekly. So I discovered Backups tab in Proxmox WebInterface, and  great vzdump tool! I just didn't realized that I need one more LV to store backup.

You just can't backup to the same Volume which You want to backup!


After getting this into mind, I quickly created backup LV (I needed to shrink existing pve/data LV):

lvcreate -L 800GB -n backup /dev/md2

After this I was able to run vzdump in snapshot mode without any fallback to suspend/stop mode. But short after creating snapshot system was unresponsive. It was a complete disaster!

Drives both on host and guest OS'es were 20x slower. My nginx reverse proxy kept showing 504 Gateway timeout errors...

How to cope with problem? There are two ways:

• add additional physical drive to PV and create backup volume only on that drive - you can still use vzdump tool
• create LV using ramdisk (for me 512MB are more than enough, I do not have lot of I/O) and set up snapshot of pve/data, copy files from snapshot to your pve/backup volume

Because, I didn't have additional HDD on my server (soft raid1 matrix /dev/md) I was left with second solution.

So, let's do this! Fistly modify Your grub to encrease /dev/ramX devices size (it will be consumed only if we put data to our volume). For me 512 MB were enough. To be sure, there will be no overflow of data I'm adding almost 1 GB.

edit /etc/default/grub that GRUB_CMDLINE_LINUX_DEFAULT will look like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet ramdisk_size=512000"

Then run update_grub ... After reboot You will have 512 MB, in each /dev/ramX .

So now backup script:

#!/bin/bash
BACKUP_ID=`date -I`
BACKUPS_TO_KEEP=6

# setup
/sbin/pvcreate /dev/ram[0-1]
/sbin/vgextend pve /dev/ram[0-1]
/sbin/lvcreate -L 984MB -s pve/data -n ramsnap /dev/ram[0-1]
mount /dev/pve/ramsnap /backup/ramsnap

# backup
mkdir /backup/vm/$BACKUP_ID
rsync -rv /backup/ramsnap/ /backup/vm/$BACKUP_ID/

# clear
umount /backup/ramsnap
/sbin/lvremove /dev/pve/ramsnap -f
/sbin/vgreduce pve /dev/ram[0-1]
/sbin/pvremove /dev/ram[0-1]

# remove old backups
BACKUPS=`ls -lt /backup/vm/ | awk 'BEGIN{} {print $9}'`
I=1
for BACKUP in $BACKUPS
do
    if [ $I -gt ${BACKUPS_TO_KEEP} ]
    then
        echo "Usuwam katalog /backup/vm/"${BACKUP}"!"
# uncomment next line to really remove it ;)
#        rm "/backup/vm/"${BACKUP} -R
    fi
    I=$((1 + $I))
done

Now I'm able to backup with speed like 20 MB/s with no I/O delays at all! Only drawback is you need to cope with unwanted/old backup folders yourself...

Hope it helps someone!