ZFS with LXD – Hardware Requirements

ZFS is using a lot of memory on servers with LXD containers. How to reduce ZFS memory usage? I will show it in this post.

1. ECC memory
2. 8GB+ of memory for the best performance
3. 2GB+ of memory (with dedup & compression off)

1. ECC memory. This isn’t really a requirement, but it’s highly recommended.
Make daily backups and make sure your data is safely stored if you don’t use ECC memory.

“There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error.” Source

2. 8GB+ of memory?
For deduplication and compression it is highly recommended to use 50 percent of the system memory. It is good to use 50 percent (default) of the system memory for ZFS ARC to prevent performance shortage of the LXD host. Can you host containers with 50% of the total memory? Don’t change the default ZFS arc settings in that case.

3. 2GB+ of memory (with dedup & compression off)
The best option for small LXD hosts/vm’s with less memory. After creating the ZFS pool you can check and turn off deduplication and compression.

Show Compression status:

zfs get compression POOLNAME

Turn off Compression:

zfs set compression=off POOLNAME

Show Dedup status:

zfs get dedup POOLNAME

Turn off Dedup:

zfs set dedup=off POOLNAME

Display the current values. (Default c_max = 50% memory)

cat /proc/spl/kstat/zfs/arcstats |grep c_

Convert bits to mbits or gbits: Open Link

Add a new file /etc/modprobe.d/zfs.conf with the new values.

# 512Mb min. memory.
options zfs zfs_arc_min=536870912
# 3Gb max. memory.
options zfs zfs_arc_max=3221225472

Save and close the file.

Reboot your system and check the new values.

cat /proc/spl/kstat/zfs/arcstats |grep c_

Monitor your configuration with:

cat /proc/spl/kstat/zfs/arcstats

That’s it for now! Thank you for reading the post.


2 thoughts on “ZFS with LXD – Hardware Requirements

  1. simon

    Hello, great article.

    I have a lxd server running eith zfs. My server has 64 GB with RAM. We arn’t running either compression or dedup. At the moment the server has started to swap, which is deed is slow.

    Could we safely reduce the filecache to 8GB? What would it cause?



Leave a Reply

Your email address will not be published.