ZFS is using a lot of memory on servers with LXD containers. How to reduce ZFS memory usage? I will show it in this post.
Recommended:
1. ECC memory
2. 8GB+ of memory for the best performance
3. 2GB+ of memory (with dedup & compression off)
1. ECC memory. This isn’t really a requirement, but it’s highly recommended.
Make daily backups and make sure your data is safely stored if you don’t use ECC memory.
“There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error.” Source
2. 8GB+ of memory?
For deduplication and compression it is highly recommended to use 50 percent of the system memory. It is good to use 50 percent (default) of the system memory for ZFS ARC to prevent performance shortage of the LXD host. Can you host containers with 50% of the total memory? Don’t change the default ZFS arc settings in that case.
3. 2GB+ of memory (with dedup & compression off)
The best option for small LXD hosts/vm’s with less memory. After creating the ZFS pool you can check and turn off deduplication and compression.
Show Compression status:
zfs get compression POOLNAMETurn off Compression:
zfs set compression=off POOLNAMEShow Dedup status:
zfs get dedup POOLNAMETurn off Dedup:
zfs set dedup=off POOLNAMEDisplay the current values. (Default c_max = 50% memory)
cat /proc/spl/kstat/zfs/arcstats |grep c_Convert bits to mbits or gbits: Open Link
Add a new file /etc/modprobe.d/zfs.conf with the new values.
# 512Mb min. memory.
options zfs zfs_arc_min=536870912
# 3Gb max. memory.
options zfs zfs_arc_max=3221225472Save and close the file.
Reboot your system and check the new values.
cat /proc/spl/kstat/zfs/arcstats |grep c_Monitor your configuration with:
cat /proc/spl/kstat/zfs/arcstatsThat’s it for now! Thank you for reading the post.
Hello, great article.
I have a lxd server running eith zfs. My server has 64 GB with RAM. We arn’t running either compression or dedup. At the moment the server has started to swap, which is deed is slow.
Could we safely reduce the filecache to 8GB? What would it cause?
Simon
Hello Simon,
Yes, it is safe in your situation to reduce the ZFS cache to 8GB with 64GB ram.
https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html (Link updated 26-05-2020)
Best Regards,
Tom