Support for Fusion IO drives?

Discussion:

(too old to reply)

Andy Young

2012-08-28 20:46:25 UTC

We are investigating adding SSDs as ZIL devices to boost our ZFS write
performance. I read an article a while ago about iX Systems teaming up with
Fusion IO to integrate their hardware with FreeBSD. Does anyone know
anything about supported drivers for Fusion IO's iodrives?

Thanks!

Andy

Josh Paetzel

2012-08-28 23:08:19 UTC

Permalink

-------- Original Message --------
Subject: Support for Fusion IO drives?
Date: Tue, 28 Aug 2012 16:46:00 -0400
We are investigating adding SSDs as ZIL devices to boost our ZFS write
performance. I read an article a while ago about iX Systems teaming up with
Fusion IO to integrate their hardware with FreeBSD. Does anyone know
anything about supported drivers for Fusion IO's iodrives?
Thanks!
Andy
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-hardware
To unsubscribe, send any mail to

I'll put on my iXsystems hat here, as well as my fast storage, ZFS and
Fusion-I/O hat.

The ZFS filesystem supports dedicated ZIL devices, which can accelerate
certain types of write requests, notably related to fsync. The VMWare
NFS client issues a sync with every write, and most databases do as
well. In those types of environments having a fast dedicated ZIL device
is almost essential. In other environments the benefits of a dedicated
ZIL range from non-existent to substantial.

A good dedicated ZIL device is all about latency. It doesn't need to be
large, in fact it will only ever handle 10 seconds of writes, so 10x
network bandwidth is worst case. (In most environments this means 20GB
is larger than needed).

Fusion-I/O cards are far too large to be cost effective ZIL devices.
Even though they do rock at I/O latency, the really fast ones are also
fairly large, so the $/GB on them isn't so attractive. There are better
options for ZIL devices.

Another consideration is the Fusion-I/O driver is fairly memory hungry,
which competes with memory ZFS wants to use for read caching.

Now as an L2ARC device, that's a whole different can of worms.

Command line used: iozone -r 4k -s 96g -i 0 -i 1 -i 2 -t 8
Parent sees throughput for 8 readers = 1712399.95 KB/sec
L2 ARC Breakdown: 197.45m
Hit Ratio: 98.61% 194.71m
L2 ARC Size: (Adaptive) 771.13 GiB
ARC Efficiency: 683.40m
Actual Hit Ratio: 71.09% 485.82m

~ 800GB test data, all served from cache.

If you are considering Fusion-I/O, the FreeBSD driver is generally not
released to the general public by Fusion-I/O, but can be obtained from
various partners. (I believe iXsystems is the only FreeBSD friendly
fusion-i/o partner but could be wrong about that)

--
Thanks,

Josh Paetzel

Andrew Young

2012-08-28 23:47:41 UTC

Permalink

Thanks for the great feedback Josh! The optimum size for an ssd zil device was still an open question for us. I'm really glad to hear that they don't need to be that big.

What does zfs do with the zil if there is no dedicated zil device? Our servers consist of a small sata drive that holds the OS and a boatload of larger drives on a sas bus. What I'm wondering is if I simply replace the OS disk with an ssd will I get the same performance boost as if I added a dedicated ssd zil?

Thanks!
Andy

Post by Josh Paetzel

I'll put on my iXsystems hat here, as well as my fast storage, ZFS and
Fusion-I/O hat.
The ZFS filesystem supports dedicated ZIL devices, which can accelerate
certain types of write requests, notably related to fsync. The VMWare
NFS client issues a sync with every write, and most databases do as
well. In those types of environments having a fast dedicated ZIL device
is almost essential. In other environments the benefits of a dedicated
ZIL range from non-existent to substantial.
A good dedicated ZIL device is all about latency. It doesn't need to be
large, in fact it will only ever handle 10 seconds of writes, so 10x
network bandwidth is worst case. (In most environments this means 20GB
is larger than needed).
Fusion-I/O cards are far too large to be cost effective ZIL devices.
Even though they do rock at I/O latency, the really fast ones are also
fairly large, so the $/GB on them isn't so attractive. There are better
options for ZIL devices.
Another consideration is the Fusion-I/O driver is fairly memory hungry,
which competes with memory ZFS wants to use for read caching.
Now as an L2ARC device, that's a whole different can of worms.
Command line used: iozone -r 4k -s 96g -i 0 -i 1 -i 2 -t 8
Parent sees throughput for 8 readers = 1712399.95 KB/sec
L2 ARC Breakdown: 197.45m
Hit Ratio: 98.61% 194.71m
L2 ARC Size: (Adaptive) 771.13 GiB
ARC Efficiency: 683.40m
Actual Hit Ratio: 71.09% 485.82m
~ 800GB test data, all served from cache.
If you are considering Fusion-I/O, the FreeBSD driver is generally not
released to the general public by Fusion-I/O, but can be obtained from
various partners. (I believe iXsystems is the only FreeBSD friendly
fusion-i/o partner but could be wrong about that)
--
Thanks,
Josh Paetzel

Nikolay Denev

2012-08-29 05:50:54 UTC

Permalink

Fron the zpool man page:

By default, the intent log is allocated from blocks within the main pool.
However, it might be possible to get better performance using separate
intent log devices such as NVRAM or a dedicated disk.

I was also contemplating the idea of a fast SSD on PCIe as a ZIL and L2ARC.
And given the fact that the SSD will not suffer from the different types and locations of IO requests,
maybe it makes sense to go with a big SSD, and partition it for a small ZIL partition,
and the rest for L2ARC. Anyone tried that?

Regards,
Nikolay

Post by Andrew Young
Thanks for the great feedback Josh! The optimum size for an ssd zil device was still an open question for us. I'm really glad to hear that they don't need to be that big.
What does zfs do with the zil if there is no dedicated zil device? Our servers consist of a small sata drive that holds the OS and a boatload of larger drives on a sas bus. What I'm wondering is if I simply replace the OS disk with an ssd will I get the same performance boost as if I added a dedicated ssd zil?
Thanks!
Andy

Post by Josh Paetzel

I'll put on my iXsystems hat here, as well as my fast storage, ZFS and
Fusion-I/O hat.
The ZFS filesystem supports dedicated ZIL devices, which can accelerate
certain types of write requests, notably related to fsync. The VMWare
NFS client issues a sync with every write, and most databases do as
well. In those types of environments having a fast dedicated ZIL device
is almost essential. In other environments the benefits of a dedicated
ZIL range from non-existent to substantial.
A good dedicated ZIL device is all about latency. It doesn't need to be
large, in fact it will only ever handle 10 seconds of writes, so 10x
network bandwidth is worst case. (In most environments this means 20GB
is larger than needed).
Fusion-I/O cards are far too large to be cost effective ZIL devices.
Even though they do rock at I/O latency, the really fast ones are also
fairly large, so the $/GB on them isn't so attractive. There are better
options for ZIL devices.
Another consideration is the Fusion-I/O driver is fairly memory hungry,
which competes with memory ZFS wants to use for read caching.
Now as an L2ARC device, that's a whole different can of worms.
Command line used: iozone -r 4k -s 96g -i 0 -i 1 -i 2 -t 8
Parent sees throughput for 8 readers = 1712399.95 KB/sec
L2 ARC Breakdown: 197.45m
Hit Ratio: 98.61% 194.71m
L2 ARC Size: (Adaptive) 771.13 GiB
ARC Efficiency: 683.40m
Actual Hit Ratio: 71.09% 485.82m
~ 800GB test data, all served from cache.
If you are considering Fusion-I/O, the FreeBSD driver is generally not
released to the general public by Fusion-I/O, but can be obtained from
various partners. (I believe iXsystems is the only FreeBSD friendly
fusion-i/o partner but could be wrong about that)
--
Thanks,
Josh Paetzel

_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-hardware