Do you recommend Supermicro AOC-SAT2-MV8 as PCI-X SATA controller?

Discussion:

(too old to reply)

Moritz Wilhelmy

2012-04-04 13:53:28 UTC

Hello,

I'm going to purchase a PCI-X SATA controller card for an NFS server.
Supermicro AOC-SAT2-MV8 looks good with it's 8 SATA connectors.

Before purchasing, and as my last inquiry, I wanted to make sure that
this particular card is well-supported by FreeBSD.

According to what I've seen in the list archives, it seems to work (with
occasional issues). Is there anything else to keep in mind?

Best regards,

Moritz

DarkSoul

2012-04-04 14:38:50 UTC

Permalink

Hello,

I used this card for my personal ZFS NAS (2 cards, 15 disks + 1 SSD).

http://www.supermicro.com/products/accessories/addon/AOC-SAT2-MV8.cfm

Frankly speaking, this card will work 99% of the time. But it has a few
quirks :
- Sometimes, I've had a port drop because maybe of a broken/dying disk.
The port would NOT recover unless I cold rebooted the server.
And a port not recovering means, even the BIOS option ROM programming
won't find it.
Quite annoying as one could guess.
- I could panic a system by removing a drive (back in the 8.1-RELEASE
days) because of thread locking/sleeping issues.
- I don't know what is to blame for that but :
- I had more than once odd queue issues and disks flapping. Not really
a problem with ZFS but VERY irritating nonetheless.
- I even had a whole controller drop on me once. Nothing a reboot/zpool
scrub couldn't fix (with NO corruption to boot!) but still...
- It really, REALLY doesn't play nice with other cards. I tried
migrating progressively to mpt(4) cards, with a one by one switch, only
to experience stray NMIs and pretty ugly kernel panics. It turned out
having a "pure" system with two f the same kind (mind, I was not pairing
PCI-X and PCIe, this was in every case pure PCI-X setups) did wonders
for stability.

It's probably fairly decent for most home purposes (my main use),
but I'd advise against it in any serious environment.

Cheers,

Post by Moritz Wilhelmy
Hello,
I'm going to purchase a PCI-X SATA controller card for an NFS server.
Supermicro AOC-SAT2-MV8 looks good with it's 8 SATA connectors.
Before purchasing, and as my last inquiry, I wanted to make sure that
this particular card is well-supported by FreeBSD.
According to what I've seen in the list archives, it seems to work (with
occasional issues). Is there anything else to keep in mind?
Best regards,
Moritz
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-hardware

--
Stephane LAPIE, EPITA SRS, Promo 2005
"Even when they have digital readouts, I can't understand them."
--MegaTokyo

Moritz Wilhelmy

2012-04-04 15:47:28 UTC

Permalink

Hello,

Post by DarkSoul
I used this card for my personal ZFS NAS (2 cards, 15 disks + 1 SSD).
http://www.supermicro.com/products/accessories/addon/AOC-SAT2-MV8.cfm
Frankly speaking, this card will work 99% of the time. But it has a few
- Sometimes, I've had a port drop because maybe of a broken/dying disk.
The port would NOT recover unless I cold rebooted the server.
And a port not recovering means, even the BIOS option ROM programming
won't find it.
Quite annoying as one could guess.
- I could panic a system by removing a drive (back in the 8.1-RELEASE
days) because of thread locking/sleeping issues.
- I had more than once odd queue issues and disks flapping. Not really
a problem with ZFS but VERY irritating nonetheless.
- I even had a whole controller drop on me once. Nothing a reboot/zpool
scrub couldn't fix (with NO corruption to boot!) but still...
- It really, REALLY doesn't play nice with other cards. I tried
migrating progressively to mpt(4) cards, with a one by one switch, only
to experience stray NMIs and pretty ugly kernel panics. It turned out
having a "pure" system with two f the same kind (mind, I was not pairing
PCI-X and PCIe, this was in every case pure PCI-X setups) did wonders
for stability.
It's probably fairly decent for most home purposes (my main use),
but I'd advise against it in any serious environment.

Can you (or someone else for that matter) recommend any decent PCI-X
controller for use in "serious environments", preferably with more than
4 Ports and in the same price category? :-)
The setup is rather serious, but then again, I don't expect having to
replace disks all the time (so if there isn't anything else that would
cost about the same, I might just go with this one)... It affects about
30 users. I heard Silicon Image cards are supposed to be good?

I don't need or want a hardware RAID-controller, because I'd prefer
using software RAID (and don't want to waste the extra money).

Best regards,

Moritz

Stephane LAPIE

2012-04-04 16:18:56 UTC

Permalink

Right now I'm using this controller :

***@pci0:6:1:0: class=0x010000 card=0x10aa1734 chip=0x00541000
rev=0x01 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS1068 PCI-X Fusion-MPT SAS'
class = mass storage
subclass = SCSI
***@pci0:6:7:0: class=0x010000 card=0x10aa1734 chip=0x00541000
rev=0x01 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS1068 PCI-X Fusion-MPT SAS'
class = mass storage
subclass = SCSI

http://www.lsi.com/products/storagecomponents/Pages/LSISAS1068.aspx

The only problem is, that card is an OEM version that was a bit tricky
to find.
I got it off an auction site in Japan, but I guess it covers pretty much
what you want,
in that it will advertise disks as physical volumes by default.

My only complaint on FreeBSD 9.X so far is that,
it has trouble doing SMART probes on the disks using passthrough :)

Performance is top notch, even though I had to tweak my /boot/loader.conf
to have a long SCSI timeout for disk detection (as you can imagine,
15 disks will take a LONG time to detect.) :
kern.cam.scsi_delay=15000

Otherwise you could pretty much shoot for this one if PCI-X is not a
requirement :

***@pci0:2:0:0: class=0x010400 card=0x92401000 chip=0x00731000
rev=0x02 hdr=0x00
vendor = 'LSI Logic (Was: Symbios Logic, NCR)'
class = mass storage
subclass = RAID

The only real beef with this one is driver support,
as I had to make it work by using LSI's driver,
(which will require some simple patching if you plan on using it on 9.X)
and it won't handle more than one card at once as it stands.
I'm not quite aware of support status for this baby in 9.0 or 10.0.

Cheers,

Post by Moritz Wilhelmy
Hello,

Can you (or someone else for that matter) recommend any decent PCI-X
controller for use in "serious environments", preferably with more than
4 Ports and in the same price category? :-)
The setup is rather serious, but then again, I don't expect having to
replace disks all the time (so if there isn't anything else that would
cost about the same, I might just go with this one)... It affects about
30 users. I heard Silicon Image cards are supposed to be good?
I don't need or want a hardware RAID-controller, because I'd prefer
using software RAID (and don't want to waste the extra money).
Best regards,
Moritz

--
Stephane LAPIE, EPITA SRS, Promo 2005
"Even when they have digital readouts, I can't understand them."
--MegaTokyo

Moritz Wilhelmy

2012-04-04 17:37:50 UTC

Permalink

Hello Stephane,

Thanks for the informative reply!

Post by Stephane LAPIE
rev=0x01 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS1068 PCI-X Fusion-MPT SAS'
class = mass storage
subclass = SCSI
rev=0x01 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS1068 PCI-X Fusion-MPT SAS'
class = mass storage
subclass = SCSI
http://www.lsi.com/products/storagecomponents/Pages/LSISAS1068.aspx
The only problem is, that card is an OEM version that was a bit tricky
to find.
I got it off an auction site in Japan, but I guess it covers pretty much
what you want,
in that it will advertise disks as physical volumes by default.

I see. I'll try to get my hands on one, but anything that's hard to buy
is also not an option (as always with universities, there is already
enough bureaucracy involved). Unfortunately, this limits the choice of
hardware :-(

Post by Stephane LAPIE
My only complaint on FreeBSD 9.X so far is that,
it has trouble doing SMART probes on the disks using passthrough :)

Uh, that's bad with that many disks... How do you work around it, if at
all?

Post by Stephane LAPIE
Performance is top notch, even though I had to tweak my /boot/loader.conf
to have a long SCSI timeout for disk detection (as you can imagine,
kern.cam.scsi_delay=15000

How's the performance of the AOC-SAT2-MV8 compared to this one?

Post by Stephane LAPIE
Otherwise you could pretty much shoot for this one if PCI-X is not a
rev=0x02 hdr=0x00
vendor = 'LSI Logic (Was: Symbios Logic, NCR)'
class = mass storage
subclass = RAID
The only real beef with this one is driver support,
as I had to make it work by using LSI's driver,
(which will require some simple patching if you plan on using it on 9.X)
and it won't handle more than one card at once as it stands.
I'm not quite aware of support status for this baby in 9.0 or 10.0.

Unfortunately, the mainboard doesn't have any PCIe slots, because then I
assume I would have a lot more cards to choose from...

Best regards,

Moritz

Moritz Wilhelmy

2012-04-04 17:44:07 UTC

Permalink

Post by Moritz Wilhelmy
How's the performance of the AOC-SAT2-MV8 compared to this one?

Silly me, the AOC-SAT2-MV8 only does 300MB/s, the LSI controller
page states that it "Provides 1.5 and 3Gb/s data transfer rates per
port".

Our current NFS server has a 3ware card, and I'm not entirely satisfied,
so I want to try something else for the new one...

Moritz

Moritz Wilhelmy

2012-04-04 18:01:21 UTC

Permalink

There must be something wrong with me today.

Post by Moritz Wilhelmy

Post by Moritz Wilhelmy
How's the performance of the AOC-SAT2-MV8 compared to this one?

Silly me, the AOC-SAT2-MV8 only does 300MB/s, the LSI controller
page states that it "Provides 1.5 and 3Gb/s data transfer rates per
port".

The AOC-SAT2-MV8 page states the following:
"Data Transfer Rates Up to 3.0 Gigabits/sec per port"
I don't know where the 300MB/s came from. Maybe the 3ware card...

How do they compare with regard to performance?

Also, sorry for the noise..

Moritz

Stephane LAPIE

2012-04-05 16:50:37 UTC

Permalink

Hello Moritz,

Post by Moritz Wilhelmy
Hello Stephane,
Thanks for the informative reply!

I see :(

Post by Moritz Wilhelmy

Post by Stephane LAPIE
My only complaint on FreeBSD 9.X so far is that,
it has trouble doing SMART probes on the disks using passthrough :)

Uh, that's bad with that many disks... How do you work around it, if at
all?

I didn't manage yet :( I'm pretty much stuck...

Post by Moritz Wilhelmy

Post by Stephane LAPIE
Performance is top notch, even though I had to tweak my /boot/loader.conf
to have a long SCSI timeout for disk detection (as you can imagine,
kern.cam.scsi_delay=15000

How's the performance of the AOC-SAT2-MV8 compared to this one?

I don't remember the exact figures. But given that the mvs(4) sometimes
hanged up (most of the time silently, really bad times would have drives
flap in and out), I say that mpt(4) comes out on top easy. Other than
that... My testing was done using all of my Gbit interfaces and trying
to max them all at once, and I could bust out something like 300-400MB/s
in pure transfer speeds using HTTP.

I'm not sure if the zpool iostat are to be trusted, but I reckon mvs(4)
would go up to 400MB/s while mpt(4) did go around 500-600MB/s... (take
that with a healthy grain of salt)

Cheers

--
Stephane LAPIE, EPITA SRS, Promo 2005
"Even when they have digital readouts, I can't understand them."
--MegaTokyo

Moritz Wilhelmy

2012-04-09 10:29:37 UTC

Permalink

Hello Stephane,

Just wondering, do you (or anyone else) know whether there are PCI-X
SATA controllers with only 4 Ports with less issues than the ones
already mentioned? :-)

Maybe 4 are enough after all...

Thanks,

Moritz

Michael Fuckner

2012-04-09 10:44:40 UTC

Permalink

Post by Moritz Wilhelmy
Hello Stephane,
Just wondering, do you (or anyone else) know whether there are PCI-X
SATA controllers with only 4 Ports with less issues than the ones
already mentioned? :-)
Maybe 4 are enough after all...

there are some with sil-chips and SAS ones like LSI 3041E-R

Regards,
Michael!

Dieter BSD

2012-04-09 20:34:06 UTC

Permalink

Post by Moritz Wilhelmy

Post by Moritz Wilhelmy
Can you (or someone else for that matter) recommend any decent PCI-X
controller for use in "serious environments"
I heard Silicon Image cards are supposed to be good?

Just wondering, do you (or anyone else) know whether there are PCI-X
SATA controllers with only 4 Ports with less issues than the ones
already mentioned? :-)
Maybe 4 are enough after all...

What do you mean by "serious environments"?

speed? reliability? harsh environment? support? other?

I have the Silicon Image 3132 which is PCIe-x1 with 2 sata ports.
Not as fast as it should be but fast enough for my needs.
Works well with FreeBSD siis(4), which provides NCQ.
Works well with the 3726 port multiplier. Talks to recent
600MB/s drives at 300MB/s, unlike JMB363 which doesn't like
600MB/s drives, even with the sata rev hint set.

Look into the 3124 which has 4 ports and is IIRC PCI-X.
Said to be faster than the 3132.

You probably want to avoid the first generation Silicon Image
sata chips. They are very slow and word is that FreeBSD doesn't
support them very well. (NetBSD does have good support, but they
are still very slow.)

If you don't need ueber speed on multiple ports at once,
port multipliers are a good way to get more ports.

Issue: if a port has a problem (flaky disk or whatever), siis(4) may
do a bunch of DELAY(big number) which interferes with other hardware
doing real-time data logging, causing data to be lost. Unacceptable.
Does not require power cycle though. I don't recall it even needing
a reboot.

Jin Guojun

2012-04-11 05:21:24 UTC

Permalink

FYI:

Sil3132 works well in general except one hardware bug for PCI read burst size.
Default is 128.
This limits the maximum SATA throughput.

As talked to Silicon Image level-2 FAE, they have not tested this feature yet
and they do not even know
how to do it. In my experience, setting this register will cause PCI-X hang or
crash.

It looks like 3132 holding the bus or generating some strange signal. We planned
to run PCI-X analyzer
on it, but this was dropped due to low priority.

-Jin

________________________________
From: Dieter BSD <***@engineer.com>
To: freebsd-***@freebsd.org
Sent: Mon, April 9, 2012 1:32:17 PM
Subject: Re: PCI-X SATA (non HW-RAID) controller recommendation

Post by Moritz Wilhelmy

Post by Moritz Wilhelmy
Can you (or someone else for that matter) recommend any decent PCI-X
controller for use in "serious environments"
I heard Silicon Image cards are supposed to be good?

Just wondering, do you (or anyone else) know whether there are PCI-X
SATA controllers with only 4 Ports with less issues than the ones
already mentioned? :-)
Maybe 4 are enough after all...

Moritz Wilhelmy

2012-04-17 11:07:38 UTC

Permalink

Hi again,

First off, sorry for the huge delay.

Post by Dieter BSD

Post by Moritz Wilhelmy

Post by Moritz Wilhelmy
Can you (or someone else for that matter) recommend any decent PCI-X
controller for use in "serious environments"
I heard Silicon Image cards are supposed to be good?

Just wondering, do you (or anyone else) know whether there are PCI-X
SATA controllers with only 4 Ports with less issues than the ones
already mentioned? :-)
Maybe 4 are enough after all...

What do you mean by "serious environments"?
speed? reliability? harsh environment? support? other?

Reliability sounds good.. So having SMART work is probably a good idea
too :-)

I won't hotplug disks all the time, so rebooting once in a while when a
disk dies should not be an issue.

This is going to be the SATA controller for an NFS Server which stores
E-Mails and User home directories. Since network will probably be the
bottleneck, speed is not as important as reliability.

Vendor-Support is not that important, as long as the card is
well-supported by FreeBSD.

I also found Exys EX-3403, which is an 8-port SATA PCI-X card with a
Marvell MV88SX6081 for about 100€, but the Exys apparently only Supports
Windows - which shouldn't matter if at least the Chip is supported by
FreeBSD, no? According to earlier posts to this list (Hello Stephane :-),
this Chip works at least on 7.2.

Can someone tell me more about it?

Post by Dieter BSD
I have the Silicon Image 3132 which is PCIe-x1 with 2 sata ports.
Not as fast as it should be but fast enough for my needs.
Works well with FreeBSD siis(4), which provides NCQ.
Works well with the 3726 port multiplier. Talks to recent
600MB/s drives at 300MB/s, unlike JMB363 which doesn't like
600MB/s drives, even with the sata rev hint set.

300MB/s should be enough.

Post by Dieter BSD
Look into the 3124 which has 4 ports and is IIRC PCI-X.
Said to be faster than the 3132.

I'll take another look around.

Post by Dieter BSD
You probably want to avoid the first generation Silicon Image
sata chips. They are very slow and word is that FreeBSD doesn't
support them very well. (NetBSD does have good support, but they
are still very slow.)

Noted, thanks for the hint.

Post by Dieter BSD
If you don't need ueber speed on multiple ports at once,
port multipliers are a good way to get more ports.

Sounds useful, thanks.

Post by Dieter BSD
Issue: if a port has a problem (flaky disk or whatever), siis(4) may
do a bunch of DELAY(big number) which interferes with other hardware
doing real-time data logging, causing data to be lost. Unacceptable.
Does not require power cycle though. I don't recall it even needing
a reboot.

That doesn't sound good, but I guess somebody can tell me the issues of
any controller or driver that comes up in this discussion. If you're OK
with siis despite this, I should too.

Sorry for being quite verbose on this list, but I believe in ruling out
bad choices before actually buying hardware ;)

Thanks,
Moritz

Dieter BSD

2012-04-17 23:02:11 UTC

Permalink

Post by Moritz Wilhelmy
having SMART work is probably a good idea

Note that some (most?) of the USB-to-SATA bridges do not provide
access to SMART, at least not on FreeBSD. Also can't turn off
the write buffer, and no NCQ, and pathetically slow.

Post by Moritz Wilhelmy

300MB/s should be enough.

With vanilla rotating drives even 150 is enough. Speed-wise, NCQ is
far more important than 300 or 600. Problem is that recent drives are
600 and don't work with JMB363. Drives used to have a jumper to pretend
to only talk 150, for controllers that didn't deal well with 300.
But as far as I know there isn't any way to get the 600 drives to
pretend to be only 150 or 300.

Post by Moritz Wilhelmy

That doesn't sound good, but I guess somebody can tell me the issues of
any controller or driver that comes up in this discussion. If you're OK
with siis despite this, I should too.

It isn't acceptable, but I've had similar/identical problems with ata(4)
and ahci(4). For all I know maybe all the disk drivers do it.
I think the problem is keeping interrupts off for too long.