Rocket Raid 622 in AHCI mode

[Mike Tancsa]

I picked up an external PMP/Cage/RR6622 combo that claims to be Sata
III. From the BIOS, the card implies it does indeed see the SATA III drives

Pic of the BIOS here
Loading Image...

ahci0: <HighPoint RocketRAID 622 AHCI SATA controller> port
0x4090-0x4097,0x4080-0x4083,0x4070-0x4077,0x4060-0x4063,0x4050-0x405f
mem 0xc2430000-0xc24307ff irq 16 at device 0.0 on pci1
ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0



However, performance is abysmal and worse than using a regular sii based
SATA II card.

It sees the PMP and cage as

pmp0 at ahcich1 bus 0 scbus1 target 15 lun 0
pmp0: <Port Multiplier 38261095 1706> ATA-0 device
pmp0: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp0: 5 fan-out ports
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD1002FAEX-00Z3A0 05.01D05> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
ada2 at ahcich1 bus 0 scbus1 target 1 lun 0
ada2: <WDC WD1002FAEX-00Z3A0 05.01D05> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad7
ada3 at ahcich1 bus 0 scbus1 target 2 lun 0
ada3: <WDC WD1502FAEX-007BA0 05.01D05> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1430799MB (2930277168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ahcich1 bus 0 scbus1 target 3 lun 0
ada4: <WDC WD2002FAEX-007BA0 05.01D05> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ahcich1 bus 0 scbus1 target 4 lun 0
ada5: <WDC WD2002FAEX-007BA0 05.01D05> ATA-8 SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

# camcontrol devlist
<WDC WD1002FAEX-00Z3A0 05.01D05> at scbus1 target 0 lun 0 (ada1,pass2)
<WDC WD1002FAEX-00Z3A0 05.01D05> at scbus1 target 1 lun 0 (ada2,pass3)
<WDC WD1502FAEX-007BA0 05.01D05> at scbus1 target 2 lun 0 (ada3,pass4)
<WDC WD2002FAEX-007BA0 05.01D05> at scbus1 target 3 lun 0 (ada4,pass5)
<WDC WD2002FAEX-007BA0 05.01D05> at scbus1 target 4 lun 0 (ada5,pass6)
<Port Multiplier 38261095 1706> at scbus1 target 15 lun 0 (pmp0,pass1)
<WDC WD5002AALX-00J37A0 15.01H15> at scbus2 target 0 lun 0 (pass0,ada0)

Writing to the individual disks seems fine, but when all the disks are
engaged, its very slow

# zpool create -f tank raidz ada1 ada2 ada3 ada4 ada5

# dd if=/dev/zero of=/tank/test1 bs=2048k count=10
10+0 records in
10+0 records out
20971520 bytes transferred in 60.370025 secs (347383 bytes/sec)
# zpool export tank; zpool import tank
# dd if=/tank/test1 of=/dev/null bs=2048k
10+0 records in
10+0 records out
20971520 bytes transferred in 7.678915 secs (2731052 bytes/sec)


# for i in `jot 5 1`; do dd if=/dev/ada$i of=/dev/null count=200
bs=2048k;done
200+0 records in
200+0 records out
419430400 bytes transferred in 3.211699 secs (130594554 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 3.034111 secs (138238327 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 3.639701 secs (115237601 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 2.841386 secs (147614716 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 3.020611 secs (138856154 bytes/sec)
#

# for i in `jot 5 1`; do dd if=/dev/zero of=/dev/ada$i count=200
bs=2048k;done
200+0 records in
200+0 records out
419430400 bytes transferred in 3.285333 secs (127667539 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 3.043410 secs (137815945 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 3.611478 secs (116138154 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 2.830386 secs (148188414 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 2.987366 secs (140401412 bytes/sec)
#

Any ideas on how to fix this controller to make it work better with ZFS
and get better speeds ?


***@pci0:1:0:0: class=0x010400 card=0x00011103 chip=0x06221103
rev=0x01 hdr=0x00
vendor = 'HighPoint Technologies, Inc.'
class = mass storage
subclass = RAID
bar [10] = type I/O Port, range 32, base 0x4090, size 8, enabled
bar [14] = type I/O Port, range 32, base 0x4080, size 4, enabled
bar [18] = type I/O Port, range 32, base 0x4070, size 8, enabled
bar [1c] = type I/O Port, range 32, base 0x4060, size 4, enabled
bar [20] = type I/O Port, range 32, base 0x4050, size 16, enabled
bar [24] = type Memory, range 32, base 0xc2430000, size 2048, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message enabled with 1 message
cap 10[70] = PCI-Express 2 legacy endpoint max data 128(512) link x1(x1)
speed 5.0(5.0)
ecap 0001[100] = AER 1 1 fatal 0 non-fatal 0 corrected


---Mike

[Alexander Motin]

Post by Mike Tancsa
I picked up an external PMP/Cage/RR6622 combo that claims to be Sata
III. From the BIOS, the card implies it does indeed see the SATA III drives
Pic of the BIOS here
http://tancsa.com/rr.jpg

I am not sure what does that number on photo mean, but I haven't even
heard about any SATA3.x port multipliers yet. SiI3826 (as I can identify
it from below) is not a new one and AFAIK only SATA 2.x.

Post by Mike Tancsa
ahci0: <HighPoint RocketRAID 622 AHCI SATA controller> port
0x4090-0x4097,0x4080-0x4083,0x4070-0x4077,0x4060-0x4063,0x4050-0x405f
mem 0xc2430000-0xc24307ff irq 16 at device 0.0 on pci1
ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0

Yes, "2 6Gbps ports, Port Multiplier supported with FBS" is what this
line of Marvell chip formally reports and it sounds great. I've tested
it quite a long ago, but IIRC while it was fast enough with direct disk
connection, it was quite slow working with port multipliers.

Post by Mike Tancsa
However, performance is abysmal and worse than using a regular sii based
SATA II card.

SiI3124 still was not beaten.

Post by Mike Tancsa
It sees the PMP and cage as
pmp0 at ahcich1 bus 0 scbus1 target 15 lun 0
pmp0: <Port Multiplier 38261095 1706> ATA-0 device
pmp0: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp0: 5 fan-out ports
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD1002FAEX-00Z3A0 05.01D05> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
ada2 at ahcich1 bus 0 scbus1 target 1 lun 0
ada2: <WDC WD1002FAEX-00Z3A0 05.01D05> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad7
ada3 at ahcich1 bus 0 scbus1 target 2 lun 0
ada3: <WDC WD1502FAEX-007BA0 05.01D05> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1430799MB (2930277168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ahcich1 bus 0 scbus1 target 3 lun 0
ada4: <WDC WD2002FAEX-007BA0 05.01D05> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ahcich1 bus 0 scbus1 target 4 lun 0
ada5: <WDC WD2002FAEX-007BA0 05.01D05> ATA-8 SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
# camcontrol devlist
<WDC WD1002FAEX-00Z3A0 05.01D05> at scbus1 target 0 lun 0 (ada1,pass2)
<WDC WD1002FAEX-00Z3A0 05.01D05> at scbus1 target 1 lun 0 (ada2,pass3)
<WDC WD1502FAEX-007BA0 05.01D05> at scbus1 target 2 lun 0 (ada3,pass4)
<WDC WD2002FAEX-007BA0 05.01D05> at scbus1 target 3 lun 0 (ada4,pass5)
<WDC WD2002FAEX-007BA0 05.01D05> at scbus1 target 4 lun 0 (ada5,pass6)
<Port Multiplier 38261095 1706> at scbus1 target 15 lun 0 (pmp0,pass1)
<WDC WD5002AALX-00J37A0 15.01H15> at scbus2 target 0 lun 0 (pass0,ada0)
Writing to the individual disks seems fine, but when all the disks are
engaged, its very slow
# zpool create -f tank raidz ada1 ada2 ada3 ada4 ada5
# dd if=/dev/zero of=/tank/test1 bs=2048k count=10
10+0 records in
10+0 records out
20971520 bytes transferred in 60.370025 secs (347383 bytes/sec)
# zpool export tank; zpool import tank
# dd if=/tank/test1 of=/dev/null bs=2048k
10+0 records in
10+0 records out
20971520 bytes transferred in 7.678915 secs (2731052 bytes/sec)

ZFS in RAIDZ does simultaneous I/O to all disks, that is probably the
most complicated case. Though I can hardly believe it can be that slow.
Even without any FBS support in HBA I would expected at least 100MB/s /
5 = 20MB/s, but definitely not 350KB/s.

Post by Mike Tancsa
# for i in `jot 5 1`; do dd if=/dev/ada$i of=/dev/null count=200
bs=2048k;done
200+0 records in
200+0 records out
419430400 bytes transferred in 3.211699 secs (130594554 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 3.034111 secs (138238327 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 3.639701 secs (115237601 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 2.841386 secs (147614716 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 3.020611 secs (138856154 bytes/sec)
#
# for i in `jot 5 1`; do dd if=/dev/zero of=/dev/ada$i count=200
bs=2048k;done
200+0 records in
200+0 records out
419430400 bytes transferred in 3.285333 secs (127667539 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 3.043410 secs (137815945 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 3.611478 secs (116138154 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 2.830386 secs (148188414 bytes/sec)
200+0 records in
200+0 records out
419430400 bytes transferred in 2.987366 secs (140401412 bytes/sec)
#
Any ideas on how to fix this controller to make it work better with ZFS
and get better speeds ?

Good question. One of several about these Marvell controllers, caused by
lack of any public documentation. :(

[Mike Tancsa]

Post by Alexander Motin

Post by Mike Tancsa
#
Any ideas on how to fix this controller to make it work better with ZFS
and get better speeds ?

Good question. One of several about these Marvell controllers, caused by
lack of any public documentation. :(

OK, thanks for responding. I figured if anyone knew definitively, it
would be you :) I will just use the 3124 for now. It has been pretty
reliably for me to make these large storage appliances.

Other than the 3124, do you recommend anything else ? I want to put
together another 20TB storage appliance. Faster is nicer, but cost is a
constraining factor.

---Mike

[Alexander Motin]

Post by Mike Tancsa

Post by Alexander Motin

Post by Mike Tancsa
#
Any ideas on how to fix this controller to make it work better with ZFS
and get better speeds ?

Good question. One of several about these Marvell controllers, caused by
lack of any public documentation. :(

OK, thanks for responding. I figured if anyone knew definitively, it
would be you :) I will just use the 3124 for now. It has been pretty
reliably for me to make these large storage appliances.
Other than the 3124, do you recommend anything else ? I want to put
together another 20TB storage appliance. Faster is nicer, but cost is a
constraining factor.

Aside from these Marvell's I don't know any other AHCI HBA with FBS
support. And HBAs without FBS are by definition are not suitable for
simultaneous access to devices on PMP.

Aside from AHCI, older Marvell's supported by mvs(4) like 88SX7042 can
do FBS, but IIRC they are also not very fast and I don't trues them much
either.

[Mike Tancsa]

Post by Alexander Motin
SiI3124 still was not beaten.

Post by Mike Tancsa
It sees the PMP and cage as
pmp0 at ahcich1 bus 0 scbus1 target 15 lun 0
pmp0: <Port Multiplier 38261095 1706> ATA-0 device
pmp0: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp0: 5 fan-out ports
Any ideas on how to fix this controller to make it work better with ZFS
and get better speeds ?

Good question. One of several about these Marvell controllers, caused by
lack of any public documentation. :(

I tried to do a little more testing to see what might be up either with
the cage, the card or the drives. It seems there is something odd about
the first slot in the cage-- I have tried 2 cages, and they both show
the same behaviour. ie. two cages (same make/model) and two of the
rocket raid cards (same make and model).

If I read/write to just the disk in slot zero, it works as expected.
However, if I create some sort of multi-disk array with the disk in the
first slot, its so slow, its almost broken.

e.g.

0{mdttestbox}# gstripe stop data
0{mdttestbox}# gstripe label -v -s 131072 data ada0 ada1
Metadata value stored on ada0.
Metadata value stored on ada1.
Done.
0{mdttestbox}# newfs -U -O2 /dev/stripe/data > /dev/null
0{mdttestbox}# mount /dev/stripe/data /mnt
0{mdttestbox}# dd if=/dev/zero of=/mnt/test bs=4096k count=100
100+0 records in
100+0 records out
419430400 bytes transferred in 56.875768 secs (7374501 bytes/sec)
0{mdttestbox}# umount /mnt;mount /dev/stripe/data /mnt
0{mdttestbox}# dd if=/mnt/test of=/dev/null bs=4096k
^C17+0 records in
17+0 records out
71303168 bytes transferred in 237.424329 secs (300320 bytes/sec)
1{mdttestbox}# gstripe stop data
gstripe: Cannot destroy device data (error=16).
1{mdttestbox}# umount /mnt ; mount /dev/stripe/data /mnt
0{mdttestbox}# umount /mnt


Yet, if I use the other 4 disks, all works well

0{mdttestbox}# gstripe stop data
0{mdttestbox}# gstripe label -v -s 131072 data ada1 ada2 ada3 ada4
Metadata value stored on ada1.
warning: ada2: only 1000204885504 bytes from 2000398933504 bytes used.
Metadata value stored on ada2.
warning: ada3: only 1000204885504 bytes from 1500301909504 bytes used.
Metadata value stored on ada3.
warning: ada4: only 1000204885504 bytes from 2000398933504 bytes used.
Metadata value stored on ada4.
Done.
0{mdttestbox}# newfs -U -O2 /dev/stripe/data > /dev/null
0{mdttestbox}# mount /dev/stripe/data /mnt
0{mdttestbox}# dd if=/dev/zero of=/mnt/test bs=4096k count=100
100+0 records in
100+0 records out
419430400 bytes transferred in 2.131533 secs (196774067 bytes/sec)
0{mdttestbox}# umount /mnt ; mount /dev/stripe/data /mnt
0{mdttestbox}# dd if=/mnt/test of=/dev/null bs=4096k
100+0 records in
100+0 records out
419430400 bytes transferred in 2.574856 secs (162894699 bytes/sec)
0{mdttestbox}#


Although its a bit odd writes are faster than reads ?


ZFS works as expected as well if I exclude the first slot.

1{mdttestbox}# zpool create stripe ada1 ada2 ada3 ada4
0{mdttestbox}# dd if=/dev/zero of=/stripe/test bs=4096k count=100
100+0 records in
100+0 records out
419430400 bytes transferred in 1.283134 secs (326879599 bytes/sec)
0{mdttestbox}# dd if=/dev/zero of=/stripe/test bs=4096k count=1000
1000+0 records in
1000+0 records out
4194304000 bytes transferred in 22.798638 secs (183971691 bytes/sec)
0{mdttestbox}# dd if=/stripe/test of=/dev/null bs=4096k
1000+0 records in
1000+0 records out
4194304000 bytes transferred in 0.472554 secs (8875820076 bytes/sec)
0{mdttestbox}# zpool export stripe
0{mdttestbox}# zpool import stripe
0{mdttestbox}# dd if=/stripe/test of=/dev/null bs=4096k
1000+0 records in
1000+0 records out
4194304000 bytes transferred in 23.068320 secs (181820956 bytes/sec)
0{mdttestbox}#

Whats odd is that the SII card does not have this problem with 5 disks
in the drive cage. Only the RR card does.

0{mdttestbox}# camcontrol devlist
<WDC WD1002FAEX-00Z3A0 05.01D05> at scbus0 target 0 lun 0 (ada0,pass1)
<WDC WD1002FAEX-00Z3A0 05.01D05> at scbus0 target 1 lun 0 (ada1,pass2)
<WDC WD1502FAEX-007BA0 05.01D05> at scbus0 target 2 lun 0 (ada3,pass4)
<WDC WD2002FAEX-007BA0 05.01D05> at scbus0 target 3 lun 0 (ada4,pass5)
<WDC WD2002FAEX-007BA0 05.01D05> at scbus0 target 4 lun 0 (ada2,pass3)
<Port Multiplier 37261095 1706> at scbus0 target 15 lun 0 (pmp0,pass0)
<WDC WD5002AALX-00J37A0 15.01H15> at scbus2 target 0 lun 0 (pass6,ada5)
0{mdttestbox}#

---Mike

[p***@puripuri.plala.or.jp]

At Tue, 28 May 2013 19:33:06 +0300,

Post by Alexander Motin
I am not sure what does that number on photo mean, but I haven't even
heard about any SATA3.x port multipliers yet. SiI3826 (as I can identify
it from below) is not a new one and AFAIK only SATA 2.x.

http://www.highpoint-tech.com/USA_new/series_r750-specifications.htm
http://www.marvell.com/storage/system-solutions/assets/Marvell-88SM97xx-PB.pdf

Marvell is shipping 88SM97xx SATA 6Gbps port multiplier chip which
can be seen on HighPoint Rocket 750 card, FYI.

[Lawrence K. Chen, P.Eng.]

That matches my experience with rr622 as well.

I replaced it with a Sil3132 card I had around, I also have a Sil3124 card in my system now.

I've been thinking of replacing the cards with ASM1061 based ones....but things have been stable, so not sure I really want to go poking around inside my computer just to try different cards. Though I have plans to drop in another SSD to use as L2ARC in the future (had picked up a Velocity Solo x1 card a while back.... )

----- Original Message -----

Post by Mike Tancsa
I picked up an external PMP/Cage/RR6622 combo that claims to be Sata
III. From the BIOS, the card implies it does indeed see the SATA III drives
Pic of the BIOS here
http://tancsa.com/rr.jpg
ahci0: <HighPoint RocketRAID 622 AHCI SATA controller> port
0x4090-0x4097,0x4080-0x4083,0x4070-0x4077,0x4060-0x4063,0x4050-0x405f
mem 0xc2430000-0xc24307ff irq 16 at device 0.0 on pci1
ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
However, performance is abysmal and worse than using a regular sii based
SATA II card.

[Mike Tancsa]

Post by Lawrence K. Chen, P.Eng.
I've been thinking of replacing the cards with ASM1061 based ones....but things have been stable, so not sure I really want to go poking around inside my computer just to try different cards. Though I have plans to drop in another SSD to use as L2ARC in the future (had picked up a Velocity Solo x1 card a while back.... )

Hi,
What does the ASM1061 attach as ? A generic AHCI controller ?

---Mike

[Lawrence K. Chen, P.Eng.]

----- Original Message -----

Post by Mike Tancsa

Post by Lawrence K. Chen, P.Eng.
I've been thinking of replacing the cards with ASM1061 based
ones....but things have been stable, so not sure I really want to
go poking around inside my computer just to try different cards.
Though I have plans to drop in another SSD to use as L2ARC in the
future (had picked up a Velocity Solo x1 card a while back.... )

Hi,
What does the ASM1061 attach as ? A generic AHCI controller ?
---Mike

I would expect so, I got the idea of trying it from looking at /usr/src/sys/dev/ahci/ahci.c