LSI 9240-4i 4K alignment

[George Kontostanos]

Hi all,

We have a server with a LSI 9240-4i controller configured in JBOD with
4 SATA disks. Running FreeBSD 9.1-Beta1:

Relevant dmesg:

FreeBSD 9.1-BETA1 #0: Thu Jul 12 09:38:51 UTC 2012
***@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.09-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
real memory = 17179869184 (16384 MB)
avail memory = 16471670784 (15708 MB)
..
mfi0: <Drake Skinny> port 0xe000-0xe0ff mem
0xf7a60000-0xf7a63fff,0xf7a00000-0xf7a3ffff irq 16 at device 0.0 on
pci1
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23
..
mfi0: 321 (397672301s/0x0020/info) - Shutdown command received from host
mfi0: 322 (boot + 3s/0x0020/info) - Firmware initialization started
(PCI ID 0073/1000/9241/1000)
mfi0: 323 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664
mfi0: 324 (boot + 3s/0x0020/info) - Firmware initialization started
(PCI ID 0073/1000/9241/1000)
mfi0: 325 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664
mfi0: 326 (boot + 5s/0x0020/info) - Package version 20.10.1-0107
mfi0: 327 (boot + 5s/0x0020/info) - Board Revision 03A
mfi0: 328 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3)
..
mfisyspd0 on mfi0
mfisyspd0: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd0: SYSPD volume attached
mfisyspd1 on mfi0
mfisyspd1: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd1: SYSPD volume attached
mfisyspd2 on mfi0
mfisyspd2: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd2: SYSPD volume attached
mfisyspd3 on mfi0
mfisyspd3: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd3: SYSPD volume attached
..
mfi0: 329 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) Info:
enclPd=ffff, scsiType=0, portMap=00,
sasAddr=4433221100000000,0000000000000000
mfi0: 330 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1)
mfi0: 331 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) Info:
enclPd=ffff, scsiType=0, portMap=02,
sasAddr=4433221102000000,0000000000000000
mfi0: 332 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2)
mfi0: 333 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) Info:
enclPd=ffff, scsiType=0, portMap=03,
sasAddr=4433221101000000,0000000000000000
mfi0: 334 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0)
mfi0: 335 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) Info:
enclPd=ffff, scsiType=0, portMap=01,
sasAddr=4433221103000000,0000000000000000
mfi0: 336 (397672376s/0x0020/info) - Time established as 08/07/12
16:32:56; (28 seconds since power on)

The problem:

When trying to create a RaidZ pool using gpart and perform a 4K
alignment using gnop, we get the follwoing error immediately after
exporting the pool and destroying the .nop devices:

id: 8043746387654554958
state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://illumos.org/msg/ZFS-8000-5E
config:

Pool FAULTED corrupted data
raidz1-0 ONLINE
13283347160590042564 UNAVAIL corrupted data
16981727992215676534 UNAVAIL corrupted data
6607570030658834339 UNAVAIL corrupted data
3435463242860701988 UNAVAIL corrupted data

When we use glabel for the same purpose with the combination of gnop,
the pool imports fine.

Any suggestions?

[George Kontostanos]

After contacting the manufacturer we got this response:

------
My apologies for the wrong information provided in my previous email.
I was under the impression that this OS is still supported but after
checking with our developer, FreeBSD is currently not supported with
the LSI Megaraid Cards due to some issue with the driver we've
provided. It will be supported in our upcoming releases which may come
by the end of this year. Please check back on our website during that
time frame for the FreeBSD driver.
Once again please accept my apologies for the inconvenience.
-----

[Steven Hartland]

----- Original Message -----
From: "George Kontostanos" <***@gmail.com>
To: "FreeBSD Stable" <freebsd-***@freebsd.org>; <freebsd-***@freebsd.org>
Sent: Thursday, August 16, 2012 10:38 AM
Subject: Re: LSI 9240-4i 4K alignment

Post by George Kontostanos
------
My apologies for the wrong information provided in my previous email.
I was under the impression that this OS is still supported but after
checking with our developer, FreeBSD is currently not supported with
the LSI Megaraid Cards due to some issue with the driver we've
provided. It will be supported in our upcoming releases which may come
by the end of this year. Please check back on our website during that
time frame for the FreeBSD driver.
Once again please accept my apologies for the inconvenience.
-----

I would ask them if they have a fix, as if they do there's nothing
stopping you building a custom kernel with the update.

Its quite common for big companies to not "officially" support until a
release but that doesn't mean they don't have a fix if you know what I
mean :)

In addition to this we're running LSI 2008 based controllers here on
8.2-RELEASE + manually merged in mps driver with ZFS just fine. They
aren't running 4k aligned but they are working well.

Regards
Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone [phone]
or return the E.mail to ***@multiplay.co.uk.

[George Kontostanos]

On Thu, Aug 16, 2012 at 2:11 PM, Steven Hartland

----- Original Message ----- From: "George Kontostanos"
Sent: Thursday, August 16, 2012 10:38 AM
Subject: Re: LSI 9240-4i 4K alignment

Post by George Kontostanos
------
My apologies for the wrong information provided in my previous email.
I was under the impression that this OS is still supported but after
checking with our developer, FreeBSD is currently not supported with
the LSI Megaraid Cards due to some issue with the driver we've
provided. It will be supported in our upcoming releases which may come
by the end of this year. Please check back on our website during that
time frame for the FreeBSD driver.
Once again please accept my apologies for the inconvenience.
-----

I would ask them if they have a fix, as if they do there's nothing
stopping you building a custom kernel with the update.

You are right but it seems more cost effective to get a new controller
that has support under the mps driver.

Its quite common for big companies to not "officially" support until a
release but that doesn't mean they don't have a fix if you know what I
mean :)

I know. The problem with that controller is that for some reason it
doesn't like gpart. I have gnoped the drives directly and it seems to
be working so far.

In addition to this we're running LSI 2008 based controllers here on
8.2-RELEASE + manually merged in mps driver with ZFS just fine. They
aren't running 4k aligned but they are working well.

Right, they fall under the mps driver like the following:

LSI Logic SAS2004 (4 Port SAS)
LSI Logic SAS2008 (8 Port SAS)
LSI Logic SAS2108 (8 Port SAS)
LSI Logic SAS2116 (16 Port SAS)
LSI Logic SAS2208 (8 Port SAS)

I have worked in the past with some of those and never had a problem.

Regards
Steve

Cheers

[Steven Hartland]

----- Original Message -----

Post by George Kontostanos
I know. The problem with that controller is that for some reason it
doesn't like gpart. I have gnoped the drives directly and it seems to
be working so far.

It still really smells like something higher up the layers than the
controller tbh.

Post by George Kontostanos

Post by Steven Hartland
In addition to this we're running LSI 2008 based controllers here on
8.2-RELEASE + manually merged in mps driver with ZFS just fine. They
aren't running 4k aligned but they are working well.

LSI Logic SAS2004 (4 Port SAS)
LSI Logic SAS2008 (8 Port SAS)
LSI Logic SAS2108 (8 Port SAS)
LSI Logic SAS2116 (16 Port SAS)
LSI Logic SAS2208 (8 Port SAS)

Confused as your 9240-4i is a SAS2008 based card so you "shouldnt" be
seeing any difference as it should be supported just fine under mps.

The cards we have here are 9211-8i if that helps.

Regards
Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone [phone]
or return the E.mail to ***@multiplay.co.uk.

[George Kontostanos]

On Thu, Aug 16, 2012 at 7:26 PM, Steven Hartland

----- Original Message ----- From: "George Kontostanos"

Post by George Kontostanos
I know. The problem with that controller is that for some reason it
doesn't like gpart. I have gnoped the drives directly and it seems to
be working so far.

It still really smells like something higher up the layers than the
controller tbh.

We have tried many combinations with different drives. Any other suggestions?

Post by George Kontostanos

Post by Steven Hartland
In addition to this we're running LSI 2008 based controllers here on
8.2-RELEASE + manually merged in mps driver with ZFS just fine. They
aren't running 4k aligned but they are working well.

LSI Logic SAS2004 (4 Port SAS)
LSI Logic SAS2008 (8 Port SAS)
LSI Logic SAS2108 (8 Port SAS)
LSI Logic SAS2116 (16 Port SAS)
LSI Logic SAS2208 (8 Port SAS)

Confused as your 9240-4i is a SAS2008 based card so you "shouldnt" be
seeing any difference as it should be supported just fine under mps.
The cards we have here are 9211-8i if that helps.

You are right, the chip specs say: LSISAS2108 RAID-on-Chip

The drives are identified as mfisyspd0, mfisyspd1, etc.

Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the
person or entity to whom it is addressed. In the event of misdirection, the
recipient is prohibited from using, copying, printing or otherwise
disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please
telephone [phone]

[Steven Hartland]

----- Original Message -----

Post by George Kontostanos

Post by Steven Hartland
It still really smells like something higher up the layers than the
controller tbh.

We have tried many combinations with different drives. Any other suggestions?

See below

Post by George Kontostanos

Post by Steven Hartland
Confused as your 9240-4i is a SAS2008 based card so you "shouldnt" be
seeing any difference as it should be supported just fine under mps.
The cards we have here are 9211-8i if that helps.

You are right, the chip specs say: LSISAS2108 RAID-on-Chip

Hmm LSI's website says its a LSISAS2008 not a LSISAS2108
http://www.lsi.com/channel/products/storagecomponents/Pages/MegaRAIDSAS9240-4i.aspx

Post by George Kontostanos
The drives are identified as mfisyspd0, mfisyspd1, etc.

Are you actually using HW RAID volumns then instead of direct disks?

If so I would advice configuring the disks as passthrough.

What does pciconf -lv say?

On boot I see the following here to 2008 chip controller:-

== /var/run/dmesg.boot ==
mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfaf3c000-0xfaf3ffff,0xfaf40000-0xfaf7ffff irq 26 at device 0.0 on pci7
mps0: Firmware: 11.00.00.00
mps0: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
mps0: [ITHREAD]

== pciconf -lv ==
***@pci0:7:0:0: class=0x010700 card=0x30201000 chip=0x00721000 rev=0x03 hdr=0x00
vendor = 'LSI Logic (Was: Symbios Logic, NCR)'
class = mass storage
subclass = SAS

I suspect reading around that the MegaRAID cards are have SAS chips on board
but are superseed by the RAID part hence mfi vs mps

I wonder if you could flash with the IT firmware instead of IR to make it show
as mps instead of mfi?

Regards
Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone [phone]
or return the E.mail to ***@multiplay.co.uk.

[Steven Hartland]

----- Original Message -----

Post by George Kontostanos
You are right, the chip specs say: LSISAS2108 RAID-on-Chip
The drives are identified as mfisyspd0, mfisyspd1, etc.

The following might be interesting to you:-
http://forums.servethehome.com/showthread.php?599-LSI-RAID-Controller-and-HBA-Complete-Listing-Plus-OEM-Models

Which states:-
LSI MegaRAID SAS 9240-4i 1x4 port internal SAS vertical,
no cache, no BBU, RAID 0, 1, 10 and 5, can be crossflashed
to LSI9211 IT/IR

This is insteresting as this is the card we're using but
in the 8 port version under mps :)

So based on that a simple firmware flash may be all you
need.

Regards
Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone [phone]
or return the E.mail to ***@multiplay.co.uk.

[George Kontostanos]

On Thu, Aug 16, 2012 at 9:51 PM, Steven Hartland

----- Original Message ----- From: "George Kontostanos"

Post by George Kontostanos
You are right, the chip specs say: LSISAS2108 RAID-on-Chip
The drives are identified as mfisyspd0, mfisyspd1, etc.

The following might be interesting to you:-
http://forums.servethehome.com/showthread.php?599-LSI-RAID-Controller-and-HBA-Complete-Listing-Plus-OEM-Models
Which states:-
LSI MegaRAID SAS 9240-4i 1x4 port internal SAS vertical,
no cache, no BBU, RAID 0, 1, 10 and 5, can be crossflashed
to LSI9211 IT/IR
This is insteresting as this is the card we're using but
in the 8 port version under mps :)
So based on that a simple firmware flash may be all you
need.
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the
person or entity to whom it is addressed. In the event of misdirection, the
recipient is prohibited from using, copying, printing or otherwise
disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please
telephone [phone]

***@pci0:1:0:0: class=0x010400 card=0x92411000 chip=0x00731000
rev=0x03 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'MegaRAID SAS 9240'
class = mass storage
subclass = RAID

The controller is configured as jbod. I will follow your advice
regarding the firmware.

Thanks for your help!

[Don Lewis]

Post by Steven Hartland
----- Original Message -----

Post by George Kontostanos
You are right, the chip specs say: LSISAS2108 RAID-on-Chip
The drives are identified as mfisyspd0, mfisyspd1, etc.

The following might be interesting to you:-
http://forums.servethehome.com/showthread.php?599-LSI-RAID-Controller-and-HBA-Complete-Listing-Plus-OEM-Models
Which states:-
LSI MegaRAID SAS 9240-4i 1x4 port internal SAS vertical,
no cache, no BBU, RAID 0, 1, 10 and 5, can be crossflashed
to LSI9211 IT/IR
This is insteresting as this is the card we're using but
in the 8 port version under mps :)

I wish I would have known this earlier. I just put together a ZFS
server using LSI MegaRAID SAS 9240-8i cards. The cabling probably would
have been cleaner with the 9211-8i, but I went with the 9240 because the
vendor that I purchased the cards from listed that 9240 as being
PCI-Express 2.0, but didn't say that about the 9211. I also got the
impression that the 9240 recognized JBOD drives with the off-the-shelf
firmware, whereas the 9211 did not.

Even LSI's own site is a bit confusing. They list the 9211 in the HBA
section, but its specs don't mention JBOD, whereas the 9240 is listed in
the RAID section and its specs do list JBOD. If the only physical
difference between the cards is the connector position, it seems odd
that they don't offer products with all the combinations of firmware and
connector position.

I haven't configured the ZFS pool yet, but I didn't have any trouble
installing FreeBSD 9.1-BETA on the GPT partitioned boot drive, which
shows up as an mfi device. I'm planning on getting the ZFS pool up and
running in the next few days.

[Steven Hartland]

----- Original Message -----
From: "Don Lewis" <***@FreeBSD.org>
To: <***@multiplay.co.uk>
Cc: <***@gmail.com>; <freebsd-***@FreeBSD.org>; <freebsd-***@FreeBSD.org>
Sent: Sunday, August 19, 2012 6:37 PM
Subject: Re: LSI 9240-4i 4K alignment

Post by Don Lewis

Post by Steven Hartland
----- Original Message -----

Post by George Kontostanos
You are right, the chip specs say: LSISAS2108 RAID-on-Chip
The drives are identified as mfisyspd0, mfisyspd1, etc.

The following might be interesting to you:-
http://forums.servethehome.com/showthread.php?599-LSI-RAID-Controller-and-HBA-Complete-Listing-Plus-OEM-Models
Which states:-
LSI MegaRAID SAS 9240-4i 1x4 port internal SAS vertical,
no cache, no BBU, RAID 0, 1, 10 and 5, can be crossflashed
to LSI9211 IT/IR
This is insteresting as this is the card we're using but
in the 8 port version under mps :)

I wish I would have known this earlier. I just put together a ZFS
server using LSI MegaRAID SAS 9240-8i cards. The cabling probably would
have been cleaner with the 9211-8i, but I went with the 9240 because the
vendor that I purchased the cards from listed that 9240 as being
PCI-Express 2.0, but didn't say that about the 9211. I also got the
impression that the 9240 recognized JBOD drives with the off-the-shelf
firmware, whereas the 9211 did not.
Even LSI's own site is a bit confusing. They list the 9211 in the HBA
section, but its specs don't mention JBOD, whereas the 9240 is listed in
the RAID section and its specs do list JBOD. If the only physical
difference between the cards is the connector position, it seems odd
that they don't offer products with all the combinations of firmware and
connector position.
I haven't configured the ZFS pool yet, but I didn't have any trouble
installing FreeBSD 9.1-BETA on the GPT partitioned boot drive, which
shows up as an mfi device. I'm planning on getting the ZFS pool up and
running in the next few days.

HBA's are the way to go if your using ZFS to manage the disks, you only
need RAID if your using a FS which doesn't manage the disk side well
such as UFS.

Its often quite common for RAID controllers to actually be slower
vs RAID controllers as the RAID stack can get in the way.

JBOD is generally what HBA's do by default which may be the reason
why LSI's site doesn't mention it.

Regards
Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone [phone]
or return the E.mail to ***@multiplay.co.uk.

[Michael Fuckner]

Post by George Kontostanos
LSI Logic SAS2004 (4 Port SAS)
LSI Logic SAS2008 (8 Port SAS)

they are used with mps driver

Post by George Kontostanos
LSI Logic SAS2108 (8 Port SAS)
LSI Logic SAS2116 (16 Port SAS)
LSI Logic SAS2208 (8 Port SAS)

LSI2308

those cards are all used with mfi.

The 9240 is special. It uses the 2008-Chip and a Megaraid-Firmware (but
no Cache and also no BBU).

Regards,
Michael!