LVM + SSD‑Cache (striped) Setup & Post‑boot Checklist

(28 TB HDD + two 500 GB SSDs)

TL;DR – Run the setup script (setup_lvm_cache.sh) once, then follow the post‑boot verification / fstab section if you want the cached LV to survive reboots.

Table of Contents

  1. Safety & prerequisites
  2. Install required packages (incl. thin‑provisioning tools)
  3. Wipe old signatures
  4. (Optional) Create a single‑disk partition on the HDD
  5. Initialise LVM Physical Volumes (PVs)
  6. Create two separate Volume Groups (VGs) – and enable early activation
  7. Create the backing LV on the HDD
  8. Build a striped cache‑pool LV on the two SSDs
  9. Attach the cache‑pool to the backing LV
  10. (Optional) Choose a cache policy – writeback vs writethrough
  11. Create a filesystem & mount it
  12. Add a permanent fstab entry (with UUID guidance)
  13. Rebuild initramfs, update GRUB and verify the stack
  14. Post‑boot troubleshooting checklist

LVM CACHE SETUP – 28 TB HDD + two 500 GB SSDs (striped)

 Devices:
   /dev/sdc1  → 28 TB HDD (backing storage)
   /dev/sda   → 500 GB SSD #1 (cache device) Raid-0
   /dev/sdb   → 500 GB SSD #2 (cache device) Raid-0

What we will end up with:

   VG_hdd      = vg_hdd          (holds the HDD PV)
   LV_data     = lv_data         (28 TB logical volume, cached)
   VG_cache    = vg_cache        (holds both SSDs)
   LV_cache    = lv_cache_pool   (striped cache pool)

0️⃣ Safety check

#!/usr/bin/env bash
set -euo pipefail          # abort on any error, treat unset vars as fatal
IFS=$'\n\t'                # split loops on newlines/tabs only
=== LVM CACHE SETUP ===
HDD (backing) : /dev/sdc1
SSD #1 (cache): /dev/sda
SSD #2 (cache): /dev/sdb
Proceed? Type YES to continue:

If you do not type YES the script exits immediately – no changes are made.

Why this matters: wiping a disk is irreversible. The prompt forces you to double‑check that the device names really correspond to your hardware.

1️⃣ Install required packages (incl. thin‑provisioning tools)

sudo apt update
sudo apt install --reinstall lvm2 thin-provisioning-tools

thin-provisioning-tools supplies the cache_check binary that LVM’s cache code needs.

2️⃣ Wipe old signatures

for dev in /dev/sdc1 /dev/sda /dev/sdb ; do
    echo "Wiping any existing LVM/partition info on $dev ..."
    sudo pvremove -ff "$dev" || true          # ignore “not a PV” errors
    sudo dmsetup remove_all 2>/dev/null || true
done

If you see Failed to find a volume group … – that’s fine; the disks are now clean.

3️⃣ (Optional) Create a single‑disk partition on the HDD

Using a GPT partition (/dev/sdc1p1) makes later repurposing easier.
The SSDs stay whole‑disk because they will be used only for cache.

# Only the large HDD needs a partition
for dev in /dev/sdc1 ; do
    echo "Creating GPT + 100 % partition on $dev ..."
    sudo parted -s "$dev" mklabel gpt \
        mkpart primary 0% 100%
done

Result: /dev/sdc1p1 appears; the script still refers to /dev/sdc1 (the kernel maps the partition as the underlying block device, so both names work).

4️⃣ Initialise LVM Physical Volumes (PVs)

sudo pvcreate /dev/sdc1   # HDD (backing)
sudo pvcreate /dev/sda    # SSD #1 – cache device
sudo pvcreate /dev/sdb    # SSD #2 – cache device

If pvcreate complains that the device is “in use”, re‑run the wipe step (pvremove) and try again.

5️⃣ Build two separate Volume Groups (VGs) – and enable early activation

# The “slow” HDD lives in its own VG
sudo vgcreate vg_hdd /dev/sdc1

# Both SSDs are placed together in a second VG that will become the cache pool
sudo vgcreate vg_cache /dev/sda /dev/sdb

# Bring both VGs up early during boot (so the cache works after a reboot)
sudo vgchange -ay --sysinit vg_hdd vg_cache

Why two VGs?
Keeping fast and slow devices isolated prevents accidental allocation of data onto the cache disks.

6️⃣ Create the backing LV on the HDD

# Allocate **all free space** in vg_hdd to a single LV called lv_data
sudo lvcreate -n lv_data -l 100%FREE vg_hdd

At this point you have a plain 28 TB logical volume that lives on the HDD.
We’ll now give it a cache.

7️⃣ Build a striped cache‑pool LV on the two SSDs

# Choose how big you want the cache to be.
# 3 TiB ≈ 10 % of the backing space (28 TB) – adjust as you wish.
CACHE_SIZE="3T"

# 1️⃣ Create a data LV that will hold the cached blocks
sudo lvcreate -n lv_cache_pool -L "$CACHE_SIZE" vg_cache

# 2️⃣ Create a tiny metadata LV (1 GiB is plenty)
sudo lvcreate -n lv_meta -L 1G vg_cache

# 3️⃣ Turn those two LVs into a *striped* cache‑pool.
#    The `--type cache-pool` command automatically stripes across the
#    underlying PVs because they belong to the same VG (vg_cache).
sudo lvconvert --type cache-pool \
        -L "$CACHE_SIZE" vg_cache/lv_cache_pool \
        --poolmetadata vg_cache/lv_meta

Result: a cache‑pool named vg_cache/lv_cache_pool that spans both SSDs.

8️⃣ Attach the cache‑pool to the backing LV

sudo lvconvert --type cache \
        -L "$CACHE_SIZE" vg_hdd/lv_data \
        --cachepool vg_cache/lv_cache_pool

After this command lvs -a will show a single logical volume (/dev/vg_hdd/lv_data) with:

  • Segtype = cache
  • Cache pool = vg_cache/lv_cache_pool

All reads/writes to /dev/vg_hdd/lv_data now go through the SSD striping layer first (write‑back by default).

9️⃣ (Optional) Choose a cache policy

Policy Behaviour
writeback (default) Fastest – writes are acknowledged once they hit the cache; dirty data is flushed to the HDD later. Good for performance, acceptable if you have a UPS or tolerate brief power‑loss risk.
writethrough Safer – every write goes straight through to the backing device before returning success. Slightly slower but eliminates “lost‑dirty‑cache” scenarios.

If you prefer writethrough, run:

sudo lvchange --cachepolicy writethrough vg_hdd/lv_data

(Leave it commented if you’re happy with the default write‑back.)

10️⃣ Create a filesystem & mount it

You can pick any Linux file system (ext4, xfs, btrfs, …).
The example below uses XFS by default – change FS_TYPE if you prefer something else.

# ----- Filesystem -------------------------------------------------
FS_TYPE="xfs"          # <-- edit to ext4, btrfs, etc. if desired

sudo mkfs.$FS_TYPE -f /dev/vg_hdd/lv_data   # force‑format the LV

# ----- Mount point -------------------------------------------------
MOUNT_POINT="/mnt/storage"
sudo mkdir -p "$MOUNT_POINT"

# Immediate mount (good for a quick test)
sudo mount /dev/vg_hdd/lv_data "$MOUNT_POINT"

11️⃣ Add a permanent fstab entry (with UUID guidance)

  1. Grab the UUID that mkfs wrote on the LV

    UUID=$(blkid -s UUID -o value /dev/vg_hdd/lv_data)

  2. Add it to /etc/fstab

    You have two convenient ways – pick whichever you prefer.

    Option A – edit with nano (or any editor) 

    sudo nano /etc/fstab

    Add a line exactly as shown, substituting the $UUID value you just obtained:

    UUID=<your‑uuid>  /mnt/storage  xfs  defaults,noatime  0 2

    Option B – one‑liner with echo 

    echo "UUID=$UUID $MOUNT_POINT $FS_TYPE defaults,noatime 0 2" | sudo tee -a /etc/fstab

Why use the UUID?
Device names (/dev/vg_hdd/lv_data, /dev/mapper/...) can change if you add/remove disks. The UUID lives inside the filesystem super‑block and never changes, so fstab will always resolve to the correct block device.

12️⃣ Rebuild initramfs, update GRUB and verify everything is working

# Update the initramfs so it contains the latest LVM metadata (including the cache devices)
sudo update-initramfs -u

# Refresh the boot loader configuration – on Ubuntu/Debian this will pick up the new initramfs
sudo update-grub

Now check that the LVM stack and the mount are correct:

# Show the full LVM layout (including cache details)
sudo lvs -a -o +devices,segtype,cache_policy,cache_used_percent

# Verify the filesystem is mounted where we expect it
df -hT "$MOUNT_POINT"

Typical output (your numbers will differ) looks like:

Filesystem                     Type   Size   Used   Avail   Use% Mounted on
/dev/mapper/vg_hdd-lv_data    xfs   28T    1.2T    26.8T   4% /mnt/storage

If the line above appears, the whole chain (PV → VG → LV → cache‑pool → filesystem) is alive and ready.

📚 References & further reading

  • LVM cache documentation – https://manpages.debian.org/lvm2-cache
  • Thin‑provisioning tools package – provides cache_check, thin_check, etc.
  • systemd‑initramfs integration – see /usr/share/initramfs-tools/scripts/local-top/lvm2 on most Debian/Ubuntu systems.

All done! Your 28 TB HDD now enjoys a fast, striped SSD cache that survives reboots and can be managed with the standard LVM tools.