5. Adding New Storage via SAN with Reference to PCMCIA and USB
“Nothing has changed on my system” is a common statement made by people calling an IT helpline for assistance. It is a well-known fact that no system remains stagnant forever, so for those who try to achieve life eternal for their systems, ultimate failure awaits. Take racing as an apt analogy. If a racer never upgrades to a newer engine (CPU) or chassis (model), then the racer will have a hard time staying competitive. Thus, in this chapter, we discuss how to add more to our “racer.”
The term storage area network (SAN) will become, if it is not already, a common one among system administrators. The capability to consolidate all storage in a data center into large frames containing many drives is indeed the direction companies will, and need to, take. Large enterprise operating systems such as HPUX, AIX, Solaris, SGI, MVS, and others have made very impressive leaps in that direction. Therefore, Linux must “Lead, follow, or get out of the way.” With vendor support from Emulex, QLogic, and others, Fibre Channel storage has become commonplace for Linux, and now system administrators must learn the tricks of the trade to become power players.
In this chapter, we discuss adding disk storage (the most commonly added item) through SAN and touch on PCMCIA/USB. We begin by defining the configuration used to demonstrate our examples and by discussing some highlights. We then discuss the addition of a PCI device to connect additional storage. Next, we move to a discussion of adding storage to a defined PCI device. Due to its complex nature, we conclude this chapter by covering a few topics with respect to adding storage through PCMCIA/USB.
Configuration
With any SAN, consideration for the balance between performance and capacity always must be paramount. If too much emphasis is placed on capacity, performance will surely suffer, so heed the word of your storage vendor about storage layout. In this chapter, we focus only on how to increase storage with a simple storage layout, and not on performance.
We use the following two servers to illustrate our concepts.
The particulars on Machine 1 are as follows:
[root@cyclops root]# uname -a
Linux cyclops 2.4.9-e.10custom-gt #4 SMP Mon Nov 1 14:17:36 EST 2004
i686 unknown
[root@cyclops root]# cat /etc/redhat-release
Red Hat Linux Advanced Server release 2.1AS (Pensacola)
Machine 2 looks like this:
atlorca4:~ # uname -a
Linux atlorca4 2.6.5-7.97-default #1 SMP Fri Jul 2 14:21:59 UTC 2004
ia64 ia64 ia64 GNU/Linux
atlorca4:~ # cat /etc/SuSE-release
SUSE LINUX Enterprise Server 9 (ia64)
VERSION = 9
The following representations outline the two servers within the SAN.
First, Machine 1 (Cyclops):
(Cyclops) Has two Emulex LP8000 HBAs
|-LP8000-|-WWN 10000000c9241327 ----> FCID 0x7b0f00 on Brocade 28000
Storage location for Brocade 28000 Domain 0x7b
Brocade 28000 Domain 0x7b port 0 -->--ISL --> Domain 0x78 Brocade 2800
Port 1
Domain 0x78 Brocade 2800 port 5 connects to >--- Storage array
|-LP8000-|-WWN 10000000c9286bda --> FCID 0x7f0d00 on Brocade 28000
Storage location for Brocade 28000 Domain 0x7f
Two ISL hopes:
Brocade 28000 Domain 0x7f port 0 -->--ISL --> Domain 0x7b Brocade 28000
port 1, then port 0 -->--ISL --> Domain 0x78 Brocade 2800 Port 1
Domain 0x78 Brocade 2800 port 5 connects to >--- Storage array
This design is nothing more than an example of fault-tolerant Host Bus Adapter (HBA). If we lose the storage port, our host loses access to the storage.
Then we have Machine 2 (atlorca4):
|-LP9802-|-WWN 10000000c93d3fc5 --> FCID 0x652d13 on Mcdata 6064 director
port 41 (2d hex = 45 dec subtract 4 = 41)
Note
This calculation works on any McDATA switch, calculating the port number from the FCID area field. Many tricks exist for different switches, but they are beyond the scope of this chapter.
As you can see in the previous example, the storage array plugs into port 41 on FCID 0x652b13 of McDATA 6064, yielding 100% locality for this host.
Kernel Module
After successful configuration is achieved, you can move on to making the device function under Linux. This is done at the driver module level. First determine whether the system can support the new hardware.
The simplest way to begin is by examining the system to determine whether another similar device is attached. See the following example of how to make this determination in a Linux environment.
Scenario 5-1: Determine Whether a Similar Device Exists
In our example, we begin by adding “another” PCI Emulex HBA LP9802 to our existing SUSE Linux Enterprise Server 9 (ia64) - Kernel 2.6.5-7.97-default machine. Again, our first step must be to confirm whether the kernel can control the newly added device.
First, determine whether a similar device exists. Begin by confirming the running OS distribution, as demonstrated next.
atlorca4:~ # cat /etc/SuSE-release
SUSE LINUX Enterprise Server 9 (ia64)
VERSION = 9
atlorca4:~ # cat /etc/lsb-release
LSB_VERSION="1.3"
Note that these files exist on almost all distributions of Linux. In Red Hat, the filename would be /etc/redhat-release. You should always check the file to confirm which operating system version is in use.
Understanding the exact OS with regards to library versions is the next step. In the next example, we depict the PCI devices with different commands to illustrate different ways to determine which PCI adapters we have installed. In the following examples, we are looking for Emulex HBAs.
atlorca4:~ # lspci
0000:00:00.0 Communication controller: Hewlett-Packard Company Auxiliary
Diva Serial Port (rev 01)
0000:00:00.1 Serial controller: Hewlett-Packard Company Diva Serial [GSP]
Multiport UART (rev 03)
0000:00:03.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010
66MHz Ultra3 SCSI Adapter (rev 01)
0000:00:03.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010
66MHz Ultra3 SCSI Adapter (rev 01)
0000:30:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010
66MHz Ultra3 SCSI Adapter (rev 01)
0000:40:01.0 PCI bridge: IBM PCI-X to PCI-X Bridge (rev 02)
0000:41:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010
66MHz Ultra3 SCSI Adapter (rev 01)
0000:41:01.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010
66MHz Ultra3 SCSI Adapter (rev 01)
0000:41:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701
Gigabit Ethernet (rev 15)
0000:50:01.0 Fibre Channel: Emulex Corporation LP9802 Fibre Channel
Adapter (rev 01)
0000:60:01.0 PCI bridge: Intel Corp. 21154 PCI-to-PCI Bridge
0000:61:04.0 USB Controller: NEC Corporation USB (rev 41)
0000:61:04.1 USB Controller: NEC Corporation USB (rev 41)
0000:61:04.2 USB Controller: NEC Corporation USB 2.0 (rev 02)
0000:61:05.0 VGA compatible controller: ATI Technologies Inc Radeon RV100
QY [Radeon 7000/VE]
0000:70:01.0 Fibre Channel: Emulex Corporation LP9802 Fibre Channel
Adapter (rev 01)
Scenario 5-2: Mapping with /proc/ioports
Now that we have confirmed that existing devices match what we are trying to add, the next logical step is checking the installed driver version. In addition, we need to confirm that a driver exists for the device that is on the PCI bus. Yes, logic dictates that if the device is present, so is the driver, but that is not always the case. The easiest way to match a hardware device to the driver is by using /proc/ioports on IA64 or by using an open source tool called lshw found at http://ezix.sourceforge.net/software/lshw.html for IA64- or IA32-based machines. Scenario 5-2 shows examples of how to make the correlation through /proc/ioports to a given device.
atlorca4:~ # cat /proc/ioports
00000000-00000fff : PCI Bus 0000:00
00000060-0000006f : i8042
00000500-000005ff : 0000:00:03.0
00000500-000005ff : sym53c8xx
00000600-000006ff : 0000:00:03.1
00000600-000006ff : sym53c8xx
00001000-00001fff : PCI Bus 0000:08
00002000-00003fff : PCI Bus 0000:10
00004000-00005fff : PCI Bus 0000:20
00006000-00007fff : PCI Bus 0000:30
00006000-000060ff : 0000:30:01.0
00006000-000060ff : sym53c8xx
00008000-00009fff : PCI Bus 0000:40
00008000-000080ff : 0000:41:01.0
00008000-000080ff : sym53c8xx
00008100-000081ff : 0000:41:01.1
00008100-000081ff : sym53c8xx
0000a000-0000bfff : PCI Bus 0000:50 ***>-Hardware address
0000a000-0000a0ff : 0000:50:01.0 ***>-Hardware address
0000a000-0000a0ff : lpfc *******>-LP9802 driver
0000c000-0000dfff : PCI Bus 0000:60
0000c000-0000c0ff : 0000:61:05.0
0000e000-0000ffff : PCI Bus 0000:70 ***>-Hardware address
0000e000-0000e0ff : 0000:70:01.0 ***>-Hardware address
0000e000-0000e0ff : lpfc *******>-LP9802 driver
Note that determining the HBA is difficult because of the lack of detailed information. However, looking at the same data through lshw yields valuable information about the HBA, as well as other features of the server. In the following example of lshw, we only present the computer description, CPU, and Emulex HBAs to save space.
atlorca4:~/lshw-B.02.03 # lshw
atlorca4
description: Computer
product: server rx7620
vendor: hp
serial: USE4415CHJ
capabilities: smbios-2.3 dmi-2.3
configuration: uuid=8785E28C-17A4-D811-964E-E240B01E4A4B
*-core
description: Motherboard
physical id: 0
*-firmware
description: BIOS
vendor: HP
physical id: 0
version: 000.018
size: 1MB
capacity: 11MB
*-memory
description: System memory
physical id: 2
size: 2016MB
*-cpu:0
product: Itanium 2
vendor: Intel Corp.
physical id: 4
bus info: cpu@0
version: 5
size: 1300MHz
*-cpu:1
product: Itanium 2
vendor: Intel Corp.
physical id: 5
bus info: cpu@1
version: 5
size: 1300MHz
capabilities: emulated hyperthreading
...
We skip a great deal to focus on the Fibre HBAs.
...
*-fiber:0
description: Fibre Channel
product: LP9802 Fibre Channel Adapter
vendor: Emulex Corporation
physical id: 8
bus info: pci@50:01.0
logical name: scsi5
version: 01
width: 64 bits
clock: 66MHz
capabilities: bus_master cap_list scsi-host
configuration: driver=lpfc
resources: iomemory:10040000-10040fff
iomemory:10041000-100410ff irq:75
*-fiber:1
description: Fibre Channel
product: LP9802 Fibre Channel Adapter
vendor: Emulex Corporation
physical id: 1
bus info: pci@70:01.0
version: 01
width: 64 bits
clock: 66MHz
capabilities: bus_master cap_list
configuration: driver=lpfc
resources: iomemory:30040000-30040fff
iomemory:30041000-300410ff irq:83
Though the previous lshw and /proc/ioports examples indicate that we have identical hardware and a driver capable of running such hardware, we have two outstanding issues. The first issue concerns the driver version, and the second concerns adding hardware when a driver or the existing hardware does not match the new hardware.
Scenario 5-3: Determine Driver Version
To address the first issue, we need to discuss methods of finding a driver for a given device. Many methods exist, but due to their speed, the following two are our favorites. The fastest way to determine the driver version is to use the cat command on the driver instance, found in /proc filesystem. In Scenario 5-2, we know that the driver is lpfc, and that it falls under the SCSI layer. In the /proc/scsi/ directory, we find that the lpfc directory contains multiple instances (HBAs), as indicated in this scenario.
atlorca4:/proc/scsi/lpfc # cat 5
Emulex LightPulse FC SCSI 2.10f
HBA: Emulex LightPulse LP9802 (2 Gigabit) on PCI bus 50 device 08 irq 75
SerialNum: P6C640BTEPZPLU
Firmware Version: 1.01A2 (H2D1.01A2)
Hdw: 2003806d
VendorId: 0xf98010df
Portname: 10:00:00:00:c9:3d:3f:c5 Nodename: 20:00:00:00:c9:3d:3f:c5
Link Up - Ready:
PortID 0x652d13
Fabric
Current speed 2G
lpfc0t00 DID 652b13 WWPN 50:06:0e:80:03:4e:46:11 WWNN
50:06:0e:80:03:4e:46:11
Note that depending on the applicable driver, manufacturer, or both, the information contained in these files can vary greatly. In the following illustration, we moved to a different host running Red Hat Linux and QLogic HBAs.
The first step to checking driver and HBA types is to determine the OS generation, as in the following example:
[root@cmtlinp3 qla2300]# cat /etc/lsb-release
LSB_VERSION="1.3"
[root@cmtlinp3 qla2300]# cat /etc/redhat-release
Red Hat Enterprise Linux ES release 3 (Taroon Update 4)
Linux cmtlinp3.atl.hp.com 2.4.21-27.ELsmp #1 SMP Wed Dec 1 21:59:02 EST
2004 i686 i686 i386 GNU/Linux
The second step is to check the driver for the QLogic cards. Note that lshw is used to find the QLogic cards. Then we go to the /proc/scsi/qla2300 directory.
[root@cmtlinp3 qla2300]# pwd
/proc/scsi/qla2300
[root@cmtlinp3 qla2300]# ll
total 0
-rw-r--r-- 1 root root 0 Apr 1 13:33 1
crw------- 1 root root 254, 0 Apr 1 13:33 HbaApiNode
[root@cmtlinp3 qla2300]# cat 0
QLogic PCI to Fibre Channel Host Adapter for QLA2340 :
Firmware version: 3.03.01, Driver version 7.01.01
Entry address = e0834060
HBA: QLA2312 , Serial# S95189
Request Queue = 0x1f8f0000, Response Queue = 0x1f8e0000
Request Queue count= 512, Response Queue count= 512
Total number of active commands = 0
Total number of interrupts = 977
Total number of IOCBs (used/max) = (0/600)
Total number of queued commands = 0
Device queue depth = 0x20
Number of free request entries = 19
Number of mailbox timeouts = 0
Number of ISP aborts = 0
Number of loop resyncs = 0
Number of retries for empty slots = 0
Number of reqs in pending_q= 0, retry_q= 0, done_q= 0, scsi_retry_q= 0
Host adapter:loop state= <READY>, flags= 0x860a13
Dpc flags = 0x1000000
MBX flags = 0x0
SRB Free Count = 4096
Link down Timeout = 008
Port down retry = 030
Login retry count = 030
Commands retried with dropped frame(s) = 0
Configured characteristic impedence: 50 ohms
Configured data rate: 1-2 Gb/sec auto-negotiate
SCSI Device Information:
scsi-qla0-adapter-node=200000e08b1c15eb;
scsi-qla0-adapter-port=210000e08b1c15eb;
scsi-qla0-target-0=50060b00001a0b12;
SCSI LUN Information:
(Id:Lun) * - indicates lun is not registered with the OS.
( 0: 0): Total reqs 298, Pending reqs 0, flags 0x0, 0:0:6c,
( 0: 1): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c,
( 0: 2): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c,
( 0: 3): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c,
( 0: 4): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c,
( 0: 5): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c,
( 0: 6): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c,
( 0: 7): Total reqs 97, Pending reqs 0, flags 0x0, 0:0:6c,
As you can see, a great deal of information can be provided, and the amount depends on the driver author. If the driver exists, as shown previously, adding the new hardware can be an easy step without taking the server offline. However, the phrase “adding hardware online” needs to be clearly defined.
As used throughout the remainder of this chapter (and as in the phrase, “adding hardware online”), “online” means not having to reboot the OS. Although this can be a great thing, note that it currently is not possible in most (if any) PCI-based machines with the Linux OS. However, adding storage to an existing PCI controller is fully allowed by almost all vendors.
To add PCI devices on the fly, first, the Processor Dependent Code (PDC) must initiate a scan to address any PCI slots that have been filled, which is done on initial power up. After the scan is complete, a wake-up command must be sent to the BIOS/PDC to reinitialize the scan procedure and power up a PCI slot. This step is followed by a rescan of the OS’s PCI bus. To reverse this process, the OS must be able to suspend any driver instance (not the entire module, which would halt all instances of the driver), allowing a PCI slot to be powered down. The capability to power down a PCI slot exists in most enterprise servers such as HPUX, Solaris, and AIX among others, but this feature has not been perfected in the same hardware platforms running Linux. Hopefully, it is just a matter of time before the systems running Linux have this capability. However, in the meantime, a reboot is needed to add a PCI device, as long as the driver exists or is noted in the /etc/sysconfig/kernel under the variable INITRD_MODULES or /etc/modules.conf.
When installing a new PCI device, it does not matter whether the driver is installed before or after the device is inserted into the PCI slot. But again, at the current time in Linux, most PCI buses do not detect a newly added PCI device without a reboot.
In the next examples, we discuss adding LUNs through the PCI bus over the HBAs. In one example, we demonstrate adding LUNs online, whereas in another example, we discuss adding LUNs when there is no existing device driver.
Adding LUNs via PCI
Many different methods of adding devices are discussed throughout this book. However, the most important method is one in which the operating system and application can stay online. In this section, we discuss ways to add LUNs to a PCI bus device on the fly (online) and offline.
In the following discussion, we shed some light on limitations with the Linux kernel SCSI source code from older kernels. The SCSI source code responsible for scanning SCSI devices is located at /usr/src/linux/drivers/scsi/scsi_scan.c and contains the code and parameters for spawning the scan on a given bus.
With older kernels such as 2.4.9-e.10, the definitions for the BLIST_ parameters were listed as follows:
#define BLIST_NOLUN 0x001 /* Don't scan for LUNs */
#define BLIST_FORCELUN 0x002 /* Known to have LUNs, force scanning */
#define BLIST_BORKEN 0x004 /* Flag for broken handshaking */
#define BLIST_KEY 0x008 /* Needs to be unlocked by special
command */
#define BLIST_SINGLELUN 0x010 /* LUNs should better not be used in
parallel */
#define BLIST_NOTQ 0x020 /* Buggy Tagged Command Queuing */
#define BLIST_SPARSELUN 0x040 /* Non consecutive LUN numbering */
#define BLIST_MAX5LUN 0x080 /* Avoid LUNS >= 5 */
#define BLIST_ISDISK 0x100 /* Treat as (removable) disk */
#define BLIST_ISROM 0x200 /* Treat as (removable) CD-ROM */
Today, with the 2.6.X kernels, the following items have been added or changed:
#define BLIST_NOLUN 0x001 /* Only scan LUN 0 */
#define BLIST_LARGELUN 0x200 /* LUNs past 7 on a SCSI-2
device */
#define BLIST_INQUIRY_36 0x400 /* override additional length
field */
#define BLIST_INQUIRY_58 0x800 /* ... for broken inquiry
responses */
#define BLIST_NOSTARTONADD 0x1000 /* do not do automatic start on
add */
#define BLIST_MS_SKIP_PAGE_08 0x2000 /* do not send ms page 0x08 */
#define BLIST_MS_SKIP_PAGE_3F 0x4000 /* do not send ms page 0x3f */
#define BLIST_USE_10_BYTE_MS 0x8000 /* use 10 byte ms before 6 byte
ms */
#define BLIST_MS_192_BYTES_FOR_3F 0x10000 /* 192 byte ms page 0x3f
request */
#define BLIST_REPORTLUN2 0x20000 /* try REPORT_LUNS even for SCSI-
2 devs
#define BLIST_NOREPORTLUN 0x40000 /* don't try REPORT_LUNS scan
(SCSI-3 devs) */
Within the previous fields, the most important thing to do if you are having issues with scanning storage is to confirm that the dev_info fields have been defined with the appropriate functions.
In the 2.4.X kernels, the dev_info was contained within the scsi_scan.c file; however, in the later kernels, the list of supported devices has become very lengthy and deserving of its own file, scsi_devinfo.c. Either way, the following example holds true for depicting device definitions according to the limits of scsi_scan.c with respect to kernel version.
Throughout the remainder of this chapter, we use an HP XP1024 or XP128 storage array, in which devices are defined as open-X, where X is a value according to the defined array group. Within the scsi_scan.c or scsi_devinfo.c file, you should find a field containing the following:
{"HP", "OPEN-", "*", BLIST_SPARSELUN}, /* HP XP Arrays */
We now know the reason why we only find LUNs 0–7 (no greater than LUN 8). It is due to the missing BLIST_LARGELUN function. To correct the problem, correct the line as follows:
{"HP", "OPEN-", "*", BLIST_SPARSELUN | BLIST_LARGELUN}, /* HP XP Arrays */
If only it were that simple. . . . Well, yes, it is that simple if the kernel source or SCSI source in the working kernel source is up to date. Unfortunately, the older Red Hat AS 2.1 server, defined in this chapter as Machine 1, is running an outdated SCSI source tree. Remember, each time that a change is made in the source, you must recompile and boot the new kernel.
Now that we have discussed some brief limitations of the scsi_scan source with respect to the device information section, let us discuss another scenario. In the following example, we refer to Machine 1, which is attached to the storage array previously mentioned. The array has only two LUNs presented through one storage port into a fabric. The host has two HBAs logging into the fabric, so it has twice the opportunity to see the devices. For simplicity, in the following example, assume that the zoning and LUN assignments are correct on the array.
In the example, we have added two LUNs with hex values 7 and 8 to the storage and proceeded to remove and add the Emulex driver to call the scsi_scan code manually. Note that this is an offline procedure because all storage, if any existed, would go offline after the driver is removed.
[root@cyclops scsi]# dmesg
The kernel reports no information.
[root@cyclops scsi]# insmod lpfcdd
Using /lib/modules/2.4.9-e.10custom-gt/kernel/drivers/scsi/lpfcdd.o
Warning: loading /lib/modules/2.4.9-e.10custom-gt/kernel/drivers/
scsi/lpfcdd.o will taint the kernel: no license
You should ignore the taint message because it is just a notification that the driver or module does not adhere to the open source guidelines.
[root@cyclops scsi]# dmesg
Emulex LightPulse FC SCSI/IP 4.20p
PCI: Enabling device 01:03.0 (0156 -> 0157)
!lpfc0:045:Vital Product Data Data: 82 23 0 36
!lpfc0:031:Link Up Event received Data: 1 1 0 0
PCI: Enabling device 01:04.0 (0156 -> 0157)
!lpfc1:031:Link Up Event received Data: 1 1 0 0
scsi2 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 18 irq 17
scsi3 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 20 irq 27
As depicted previously, no LUNs were found. Next, we could proceed to determine whether this is a fabric zoning issue or an array LUN assignment problem. Note that in the previous test, however, we are trying to depict a true code limitation.
With Machine 1, we can depict the limitation in the scsi_scan.c source code by reviewing the definitions set forth in the code. An example of this limitation follows:
#define BLIST_NOLUN 0x001 /* Don't scan for LUNs */
#define BLIST_FORCELUN 0x002 /* Known to have LUNs, force
scanning */
#define BLIST_BORKEN 0x004 /* Flag for broken handshaking */
#define BLIST_KEY 0x008 /* Needs to be unlocked by special
command */
#define BLIST_SINGLELUN 0x010 /* LUNs should better not be used
in parallel */
#define BLIST_NOTQ 0x020 /* Buggy Tagged Command Queuing */
#define BLIST_SPARSELUN 0x040 /* Non consecutive LUN numbering */
#define BLIST_MAX5LUN 0x080 /* Avoid LUNS >= 5 */
#define BLIST_ISDISK 0x100 /* Treat as (removable) disk */
#define BLIST_ISROM 0x200 /* Treat as (removable) CD-ROM */
Note that BLIST_LARGELUN does not exist because it was added some time around the 2.4-21 kernel. BLIST_LARGELUN allows for device scanning over seven LUNs. However, the BLIST_SPARSELUN exists but did not help us find our sparse LUNs because LUN 0 must exist to start scanning the bus. Some drivers build a pseudo LUN 0 so that this condition never exists; however, most system administrators argue that skipping LUN 0 is not helpful. Nevertheless, LUN 0 does not have to exist if different scanning software is implemented.
By using scsidev-2.35 scsidev -l, we can bypass this limitation and find our disk on the fly. At this point in the process, keeping track of the devices becomes substantially more difficult.
In the following example, only Hex LUN value 8 exists on the array. Because LUN 0 is not defined, the BLIST_SPARSELUN is not called, so dmesg reports no new disk.
After you issue the insmod lpfcdd commands, the following results are found in dmesg.
Emulex LightPulse FC SCSI/IP 4.20p
PCI: Enabling device 01:03.0 (0156 -> 0157)
!lpfc0:045:Vital Product Data Data: 82 23 0 36
!lpfc0:031:Link Up Event received Data: 1 1 0 0
PCI: Enabling device 01:04.0 (0156 -> 0157)
!lpfc1:031:Link Up Event received Data: 1 1 0 0
scsi2 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 18 irq 17
scsi3 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 20 irq 27
Again, no LUNs were found. Even by running the following command, we cannot get around this issue because it uses the same scsi_scan.c code.
[root@cyclops scsi]# /tmp/scsidev-2.35/scsidev
/proc/scsi/scsi extensions not found. Fall back to scanning.
After the full scan starts, scsidev creates a symbolic device pointer that points to the internal SCSI disk /dev/sda. Note, however, that we do not need or use this device link in our servers. We have truncated a lot just to show the external SCSI disk.
Found /dev/scsi/sgh0-2800c0i0l0 (Type 00) on sym53c8xx-1.7.3c-20010512
Found /dev/scsi/sgh0-2800c0i1l0 (Type 00) on sym53c8xx-1.7.3c-20010512
Found /dev/scsi/sgh0-2800c0i2l0 (Type 00) on sym53c8xx-1.7.3c-20010512
Found /dev/scsi/sgh0-2800c0i3l0 (Type 00) on sym53c8xx-1.7.3c-20010512
Found /dev/scsi/sgh0-2800c0i4l0 (Type 00) on sym53c8xx-1.7.3c-20010512
Found /dev/scsi/sgh0-2800c0i5l0 (Type 00) on sym53c8xx-1.7.3c-20010512
Found /dev/scsi/sgh0-2800c0i11l0 (Type 03) on sym53c8xx-1.7.3c-20010512
Found /dev/scsi/sgh0-2800c0i15l0 (Type 03) on sym53c8xx-1.7.3c-20010512
However, note that by adding a LUN 0 to the array, we can get around this problem, as indicated next.
[root@cyclops scsi]# rmmod lpfcdd
[root@cyclops scsi]# insmod lpfcdd
[root@cyclops scsi]# dmesg
...
Emulex LightPulse FC SCSI/IP 4.20p
PCI: Enabling device 01:03.0 (0156 -> 0157)
!lpfc0:045:Vital Product Data Data: 82 23 0 36
!lpfc0:031:Link Up Event received Data: 1 1 0 0
PCI: Enabling device 01:04.0 (0156 -> 0157)
!lpfc1:031:Link Up Event received Data: 1 1 0 0
scsi2 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 18 irq 17
scsi3 : Emulex LPFC (LP8000) SCSI on PCI bus 01 device 20 irq 27
Vendor: HP Model: OPEN-9-CVS Rev: 2111
Type: Direct-Access ANSI SCSI revision: 02
Vendor: HP Model: OPEN-9-CVS Rev: 2111
Type: Direct-Access ANSI SCSI revision: 02
Vendor: HP Model: OPEN-9-CVS Rev: 2111
Type: Direct-Access ANSI SCSI revision: 02
Vendor: HP Model: OPEN-9-CVS Rev: 2111
Type: Direct-Access ANSI SCSI revision: 02
Vendor: HP Model: OPEN-9-CVS Rev: 2111
Type: Direct-Access ANSI SCSI revision: 02
Vendor: HP Model: OPEN-9-CVS Rev: 2111
Type: Direct-Access ANSI SCSI revision: 02
Attached scsi disk sdg at scsi2, channel 0, id 0, lun 0
Attached scsi disk sdh at scsi2, channel 0, id 0, lun 1
Attached scsi disk sdi at scsi2, channel 0, id 0, lun 10
Attached scsi disk sdj at scsi3, channel 0, id 0, lun 0
Attached scsi disk sdk at scsi3, channel 0, id 0, lun 1
Attached scsi disk sdl at scsi3, channel 0, id 0, lun 10
SCSI device sdg: 103680 512-byte hdwr sectors (53 MB)
sdg: unknown partition table
SCSI device sdh: 103680 512-byte hdwr sectors (53 MB)
sdh: unknown partition table
SCSI device sdi: 103680 512-byte hdwr sectors (53 MB)
sdi: unknown partition table
SCSI device sdj: 103680 512-byte hdwr sectors (53 MB)
sdj: unknown partition table
SCSI device sdk: 103680 512-byte hdwr sectors (53 MB)
sdk: unknown partition table
SCSI device sdl: 103680 512-byte hdwr sectors (53 MB)
sdl: unknown partition table
Having a LUN 0 or sparse LUNs is dependent on the driver and the parameters set forth in the scsi_scan.c file. Though each setting has a large impact, no impact is as great as when a single LUN is added to the PCI bus, which causes applications to crash. The next example demonstrates how a single LUN can have such an enormous impact.
We first must cover some basics. Again in this example, we use a few LUNs from a storage array. The LUN values are 0, 1, 2, 3, 10, and 11. There is no particular reason for the LUN values to be scattered as such, but any system administrator will run into this configuration at some point. With this setup, the user needs to be made aware of several issues that must be handled appropriately. For example, one issue is to determine how many targets an HBA sees.
[root@cyclops tmp]# cat /proc/scsi/lpfc/2
Emulex LightPulse LPFC Driver Version: 2.01g
HBA: Emulex LightPulse LP8000 1 Gigabit PCI Fibre Channel Adapter
SerialNum: 2R04714316
Firmware Version: 3.82A1 (D2D3.82A1)
Hdw: 2002506d
VendorId: 0xf80010df
Portname: 10:00:00:00:c9:24:13:27 Nodename: 20:00:00:00:c9:24:13:27
Link Up - Ready:
PortID 0x7b0f00
Fabric
Current speed 1G
lpfc0t00 DID 770d00 WWPN 50:00:60:e8:02:ea:81:00 WWNN
50:00:60:e8:02:ea:81:00 <---Target is an XP48 port CL1-A
lpfc0t01 DID 780500 WWPN 50:06:0e:80:03:4e:46:05 WWNN
50:06:0e:80:03:4e:46:05 <---Target is an XP128 port CL1-F
Do not forget to look at all targets presented by every HBA, as continued next:
[root@cyclops tmp]# cat /proc/scsi/lpfc/3
Emulex LightPulse LPFC Driver Version: 2.01g
HBA: Emulex LightPulse LP8000 1 Gigabit PCI Fibre Channel Adapter
SerialNum: 0000c9286bda
Firmware Version: 3.90A7 (D2D3.90A7)
Hdw: 2002506d
VendorId: 0xf80010df
Portname: 10:00:00:00:c9:28:6b:da Nodename: 20:00:00:00:c9:28:6b:da
Link Up - Ready:
PortID 0x7f0d00
Fabric
Current speed 1G
lpfc1t00 DID 770d00 WWPN 50:00:60:e8:02:ea:81:00 WWNN
50:00:60:e8:02:ea:81:00 <---Target is an XP48 port CL1-A
lpfc1t01 DID 780500 WWPN 50:06:0e:80:03:4e:46:05 WWNN
50:06:0e:80:03:4e:46:05 <---Target is an XP128 port CL1-F
Though this is not a true dual-path configuration (because it has only a single storage path), it still is a good illustration of the following fault. As shown previously, the HBA in question must be connected to a fabric because of 24-bit FCID 0x7b0f00. However, the dmesg and /var/log/ messages only show LUNs from a single target.
Attached scsi disk sdg at scsi2, channel 0, id 1, lun 0
Attached scsi disk sdh at scsi2, channel 0, id 1, lun 1
Attached scsi disk sdi at scsi2, channel 0, id 1, lun 2
Attached scsi disk sdj at scsi2, channel 0, id 1, lun 3
Attached scsi disk sdk at scsi2, channel 0, id 1, lun 10
Attached scsi disk sdl at scsi2, channel 0, id 1, lun 11
Attached scsi disk sdm at scsi3, channel 0, id 1, lun 0
Attached scsi disk sdn at scsi3, channel 0, id 1, lun 1
Attached scsi disk sdo at scsi3, channel 0, id 1, lun 2
Attached scsi disk sdp at scsi3, channel 0, id 1, lun 3
Attached scsi disk sdq at scsi3, channel 0, id 1, lun 10
Attached scsi disk sdr at scsi3, channel 0, id 1, lun 11
All the previous LUNs are on target 1 rather than target 0. After adding a few LUNs on target 0, we simply need to run the following command:
echo "scsi add-single-device 2 0 00 01" >/proc/scsi/scsi
Upon reviewing dmesg, we find that the device was added and bound to /dev/sds.
scsi singledevice 2 0 0 1
blk: queue dfb84a1c, I/O limit 4095Mb (mask 0xffffffff)
Vendor: HP Model: OPEN-E-CM Rev: 0119
Type: Direct-Access ANSI SCSI revision: 03
blk: queue dfb8461c, I/O limit 4095Mb (mask 0xffffffff)
Attached scsi disk sds at scsi2, channel 0, id 0, lun 1
SCSI device sds: 28452960 512-byte hdwr sectors (14568 MB)
sds:
After careful review, we add two other disks, LUNs 6 and 2, in that order:
echo "scsi add-single-device 2 0 00 06" >/proc/scsi/scsi
dmesg prints the following for LUN 6:
scsi singledevice 2 0 0 6
blk: queue e5c65c1c, I/O limit 4095Mb (mask 0xffffffff)
Vendor: HP Model: OPEN-3*2 Rev: 0119
Type: Direct-Access ANSI SCSI revision: 03
blk: queue e5c65e1c, I/O limit 4095Mb (mask 0xffffffff)
Attached scsi disk sdt at scsi2, channel 0, id 0, lun 6
SCSI device sdt: 9613440 512-byte hdwr sectors (4922 MB)
sdt: unknown partition table
echo "scsi add-single-device 2 0 00 02" >/proc/scsi/scsi
dmesg prints the following for LUN 2:
scsi singledevice 2 0 0 2
blk: queue e5c6501c, I/O limit 4095Mb (mask 0xffffffff)
Vendor: HP Model: OPEN-E Rev: 0119
Type: Direct-Access ANSI SCSI revision: 03
blk: queue e5c6521c, I/O limit 4095Mb (mask 0xffffffff)
Attached scsi disk sdu at scsi2, channel 0, id 0, lun 2
SCSI device sdu: 28452960 512-byte hdwr sectors (14568 MB)
sdu: unknown partition table
The new list of devices is as follows:
Attached scsi disk sdt at scsi3, channel 0, id 0, lun 1
Attached scsi disk sdg at scsi2, channel 0, id 1, lun 0
Attached scsi disk sdh at scsi2, channel 0, id 1, lun 1
Attached scsi disk sdi at scsi2, channel 0, id 1, lun 2
Attached scsi disk sdj at scsi2, channel 0, id 1, lun 3
Attached scsi disk sdk at scsi2, channel 0, id 1, lun 10
Attached scsi disk sdl at scsi2, channel 0, id 1, lun 11
Attached scsi disk sdm at scsi3, channel 0, id 1, lun 0
Attached scsi disk sdn at scsi3, channel 0, id 1, lun 1
Attached scsi disk sdo at scsi3, channel 0, id 1, lun 2
Attached scsi disk sdp at scsi3, channel 0, id 1, lun 3
Attached scsi disk sdq at scsi3, channel 0, id 1, lun 10
Attached scsi disk sdr at scsi3, channel 0, id 1, lun 11
Attached scsi disk sds at scsi2, channel 0, id 0, lun 1
Attached scsi disk sdt at scsi2, channel 0, id 0, lun 6
Attached scsi disk sdu at scsi2, channel 0, id 0, lun 2
The biggest problem is with LUNs 6 and 2 at the bottom of the list. The next time that the host is rebooted, the LUNs swap sdt and sdu values to the hard disk map point. This type of problem is why Linux developers created the scsidev project.
After running scsidev, /dev/scsi block devices are built and listed by the SCSI interface, channel, ID, and LUN value as follows:
brw-rw---- 1 root disk 65, 32 Apr 10 20:13 sdh2-0c0i0l1
brw-rw---- 1 root disk 65, 64 Apr 10 20:13 sdh2-0c0i0l2
brw-rw---- 1 root disk 65, 48 Apr 10 20:13 sdh2-0c0i0l6
brw-rw---- 1 root disk 8, 96 Apr 10 20:16 sdh2-0c0i1l0
brw-rw---- 1 root disk 8, 112 Apr 10 20:16 sdh2-0c0i1l1
brw-rw---- 1 root disk 8, 160 Apr 10 20:16 sdh2-0c0i1l10
brw-rw---- 1 root disk 8, 176 Apr 10 20:16 sdh2-0c0i1l11
brw-rw---- 1 root disk 8, 128 Apr 10 20:16 sdh2-0c0i1l2
brw-rw---- 1 root disk 8, 144 Apr 10 20:16 sdh2-0c0i1l3
brw-rw---- 1 root disk 8, 192 Apr 10 20:16 sdh3-1c0i1l0
brw-rw---- 1 root disk 8, 208 Apr 10 20:16 sdh3-1c0i1l1
brw-rw---- 1 root disk 65, 0 Apr 10 20:16 sdh3-1c0i1l10
brw-rw---- 1 root disk 65, 16 Apr 10 20:13 sdh3-1c0i1l11
brw-rw---- 1 root disk 8, 224 Apr 10 20:16 sdh3-1c0i1l2
brw-rw---- 1 root disk 8, 240 Apr 10 20:16 sdh3-1c0i1l3
Of course, modify the /etc/fstab accordingly so that if the /dev/sdX changes after a reboot, the /dev/scsi/sdhX-XcXiXlX does not.
The new LUN has been found at /dev/scsi/sgh2-0c0i0l0, and a symbolic link has been made at /dev/sdq with the alternate path on /dev/sdj for block control. The nice feature of using scsidev to find new devices is that you don’t have to worry about the LUN order.
Think of it this way. A deck of cards has been stacked neatly on a table. If you remove the card on the bottom of the stack, the entire deck shifts down. In normal circumstances, this is of no concern. However, let’s assume that a number has been written upon each card. The cards are numbered in order from 0 to 52, starting from the bottom. If you remove the number two card, does the number three card become number two? No, the value written on the card stays the same. That is the function that scsidev provides. Using the standard device addressing in Linux, the number three card becomes number two when the number two card is removed, which causes a great mess in /etc/fstab and any user application hardbound to a SCSI device file.
Adding Storage via PCMCIA/USB
Other methods of adding storage include USB, FireWire, and PCMCIA devices, and similar troubleshooting techniques are used for each. The unique aspect of PCMCIA storage is how the drivers are called with regard to sharing a single interrupt for multiple function devices. Among PCMCIA devices, the most common device type is a modem or Ethernet combo card (storage type PCMCIA cards are very rare). Note that combo cards share a single interrupt. In the following example, a single-function PCMCIA card illustrates the drivers that are loaded after the PCMCIA core finds the newly added device.
- Collect
lsmodso that a pre-driver list can be collected. - Insert PCMCIA storage device with IDE interface connecting a 5200-RPM laptop drive.
- Run
lsmodso that a post-driver list can be collected. - Run
dmesgto view the actions taken by the kernel.
When you run dmesg, it reports:
cs: memory probe 0xa0000000-0xa0ffffff: excluding 0xa0000000-0xa01fffff
Probing IDE interface ide2...
hde: HITACHI_DK239A-65B, ATA DISK drive
ide2 at 0x180-0x187,0x386 on irq 3
hde: max request size: 128KiB
hde: 12594960 sectors (6448 MB) w/512KiB Cache, CHS=13328/15/63
hde: cache flushes not supported
hde: hde1 hde2 < hde5 hde6 > hde3 hde4
ide-cs: hde: Vcc = 5.0, Vpp = 0.0
The lsmod output reflects that the kernel module ide_cs was installed, which bound to modules ds, yenta_socket, and pcmcia_core.
This is what you see before the driver installation:
ds 17796 6
yenta_socket 19840 0
pcmcia_core 66100 2 ds,yenta_socket
This is what you see after the driver installation:
ide_cs 7556 1
ds 17796 7 ide_cs
yenta_socket 19840 1
pcmcia_core 66100 3 ide_cs,ds,yenta_socket
But how does the correct driver become installed? To answer this question, we look at the PCMCIA core definitions located in directory /etc/pcmcia/. In this directory, a configuration file exists called config.
Within this file, sections exist such as the following:
card "ATA/IDE Fixed Disk"
function fixed_disk
bind "ide-cs"
The sections define how the newly attached device should be handled with respect to the driver. Within this same directory, another file exists that allows certain parameters always to be in play when a newly added PCMCIA device is attached using only IDE. The file is called ide.opts and contains options such as the following:
#INFO="Sample IDE setup"
#DO_FSTAB="y" ; DO_FSCK="y" ; DO_MOUNT="y"
#FSTYPE="msdos"
#OPTS=""
#MOUNTPT="/mnt/ide"
Some of the ATA/IDE fixed-disk device parameters follow:
• DO_FSTAB—A boolean (y/n) setting that specifies whether an entry should be added to /etc/fstab for this device.
• DO_FSCK—A boolean (y/n) setting that specifies whether the filesystem should be checked before being mounted with fsck -Ta'.
• DO_MOUNT—A boolean (y/n) setting that specifies whether this device should be automatically mounted at card insertion time.
• FSTYPE OPTS MOUNTPT—The filesystem type mount options and mount point to be used for the fstab entry and/or mounting the device.
The following example of an ide.opts entry demonstrates the first partition of any ATA/IDE card with an XFS filesystem to be mounted on /mnt.
case "$ADDRESS" in
*,*,*,1)
#INFO="Sample IDE setup"
DO_FSTAB="y" ; DO_FSCK="n" ; DO_MOUNT="y"
#FSTYPE="xfs"
#OPTS=""
#MOUNTPT="/mnt"
;;
*,*,*)
#PARTS="1"
# Card eject policy options
NO_CHECK=n
NO_FUSER=n
;;
esac
With any new Linux machine, adding PCMCIA storage is second nature; however; on the special occasion in which we find ourselves with an older IDE device and kernel, we must remember that spin-up times vary. Longer spin-up times can surpass the maximum allowed card setup time. A simple workaround for this feature, “a positive spin on a bad issue,” was developed starting around release 3.0.6 of the ide_cs driver to automatically retry the device probe. This workaround provides any device with ample time to initialize. However, if you find yourself with an even older version of ide_cs, load the pcmcia_core module with the following:
CORE_OPTS="unreset_delay=400"
For those who are trying to use a PCMCIA ATA/IDE CD-ROM device, your kernel must be compiled with CONFIG_BLK_DEV_IDECD enabled. Again, this is the default for almost all newer kernels; however, it is something to remember when developing a new custom kernel.
Summary
Though we can add storage through many methods, the particular methods discussed in this chapter can be characterized as both “common” and “troublesome.” They are “common” because SANs have become common in enterprise environments. They are “troublesome” because with the capability to handle multiple interrupts, some PCMCIA hardware vendors make device support difficult. We hope the troubleshooting steps discussed in this chapter shed a little light on both SAN and PCMCIA storage devices.