Fibre Channel over Ethernet

LIO Target
	Cisco Systems; FCoE fabric module
Original author(s)	Joe Eykholt; Kiran Patil
Developer(s)	Cisco Systems, Inc.
Initial release	July 21, 2011
Stable release	4.1.0 / June 20, 2012; ; 10 months ago
Preview release	4.2.0-rc5 / June 28, 2012; ; 10 months ago
Development status	Production
Written in	C
Operating system	Linux
Type	Fabric module
License	GNU General Public License
Website	www.cisco.com

The Linux SCSI Target Wiki

Revision as of 06:58, 29 February 2012 by Marcf (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Jump to: navigation, search

See Target for a complete overview over all fabric modules.

The Fibre Channel over Ethernet (FCoE) fabric module (tcm_fc.ko) provides Fibre Channel (FC) access over Ethernet. It is based on a fabric module from open-fcoe.org. The Open-FCoE target code has been adapted to use a modified upstream Linux libfc module, and was incorporated on 3/19/2010.

This includes a patch series to reenable point-to-point module for libfc (merged upstream), and a set of hooks into libfc to be able to hand off incoming FLOGI requests to Fibre Channel based Target modules.

The FCoE fabric module went upstream into the Linux 3.0 kernel on 5/17/2011.^[1]

Specifications

The full advanced Target SPC-3/SPC-4 SCSI logic (such as Persistent_Reservations, ALUA, etc.) is inherently working on FCoE Ports and (across) all other fabric modules, such as iSCSI, Fibre Channel and tcm_loop.

The following specifications are available from the T11 Working Group:

Fibre Channel - Backbone (FC-BB): to develop the mappings necessary to bridge between physically-separate instances of the same network definition, including MAC address mapping & translation, configuration discovery, management facilities and mappings of FC Service definitions. Candidate network definitions are all flavors of Ethernet (incl. Gigabit) & FDDI, but will be a function of resources provided. Murali Rajagopal (edt.), Status: Published, 8/1/1996

Setup

First, load the FCoE module and create an association to the matching LPORT interface configured with tcm_fc:

modprobe fcoe ; echo eth0 > /sys/module/fcoe/parameters/create

You should see something like the following in the target side kernel ring buffer:

[  469.432122] device eth0 entered promiscuous mode
[  469.433179] scsi0 : FCoE Driver
[  469.435336] host0: libfc: Link up on port (     0)

Then the FCoE fabric module can be configured by using targetcli, as described in FCoE/targetcli.

The FCoE fabric module can also be configured directly with ConfigFS via FCoE/configFS.

Initiator

When the Target is configured, we are ready to connect from an Open-FCoE Initiator. Remember that in order to run the Initiator in point-to-point mode, the necessary patches to libfc are also required. Just as on the target side, load the FCoE module and create the association to the network interface on the initiator capable of communication with the LPORT network interface configured above:

modprobe fcoe ; echo eth0 > /sys/module/fcoe/parameters/create

After a few seconds, the FCoE LUNs will be detected and registered with Linux/SCSI. Here is what that output looks like on the Open-FCOE initiator side:

[  853.183484] device eth0 entered promiscuous mode
[  853.184640] scsi0 : FCoE Driver
[  853.186064] host0: libfc: Link up on port (     0)
[  856.754764] host0: Assigned Port ID  10101
[  856.758376] scsi 0:0:0:0: Direct-Access     LIO-ORG  FILEIO           3.1  PQ: 0 ANSI: 5
[  856.818511] sd 0:0:0:0: [sda] READ CAPACITY(16) failed
[  856.819201] sd 0:0:0:0: [sda] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[  856.820127] sd 0:0:0:0: [sda] Sense not available.
[  856.820964] sd 0:0:0:0: [sda] 97656251 512-byte logical blocks: (50.0 GB/46.5 GiB)
[  856.822580] sd 0:0:0:0: [sda] Write Protect is off
[  856.823230] sd 0:0:0:0: [sda] Mode Sense: 2f 00 00 00
[  856.823640] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  856.825018] sd 0:0:0:0: [sda] READ CAPACITY(16) failed
[  856.825688] sd 0:0:0:0: [sda] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[  856.826620] sd 0:0:0:0: [sda] Sense not available.
[  856.828304]  sda: unknown partition table
[  856.830403] sd 0:0:0:0: [sda] READ CAPACITY(16) failed
[  856.831109] sd 0:0:0:0: [sda] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[  856.832039] sd 0:0:0:0: [sda] Sense not available.
[  856.833689] sd 0:0:0:0: [sda] Attached SCSI disk
[  857.186393] ------------[ cut here ]------------
[  857.187054] WARNING: at drivers/scsi/libfc/fc_lport.c:1370 fc_lport_timeout+0x49/0x87 [libfc]()
[  857.188144] Hardware name: 
[  857.191292] Modules linked in: sd_mod crc_t10dif fcoe libfcoe libfc scsi_transport_fc scsi_tgt ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod scsi_dh loop parport_pc serio_raw parport tpm_tis virtio_balloon psmouse tpm tpm_bios pcspkr i2c_piix4 i2c_core processor button evdev ext3 jbd mbcache ide_cd_mod   ide_gd_mod cdrom ata_piix ata_generic libata scsi_mod piix ide_pci_generic floppy e1000e virtio_pci virtio_ring virtio 8139cp 8139too ide_core mii thermal fan thermal_sys
[  857.202922] Pid: 16, comm: events/1 Not tainted 2.6.34-rc1 #3
[  857.203563] Call Trace:
[  857.204064]  [<ffffffffa03594e1>] ? fc_lport_timeout+0x49/0x87 [libfc]
[  857.204747]  [<ffffffffa03594e1>] ? fc_lport_timeout+0x49/0x87 [libfc]
[  857.205434]  [<ffffffff8103892d>] ? warn_slowpath_common+0x77/0xa3
[  857.206102]  [<ffffffffa03594e1>] ? fc_lport_timeout+0x49/0x87 [libfc]
[  857.206878]  [<ffffffff8104cb06>] ? worker_thread+0x181/0x20f
[  857.207557]  [<ffffffffa0359498>] ? fc_lport_timeout+0x0/0x87 [libfc]
[  857.208285]  [<ffffffff81050126>] ? autoremove_wake_function+0x0/0x2e
[  857.209009]  [<ffffffff8104c985>] ? worker_thread+0x0/0x20f
[  857.209693]  [<ffffffff8104fcf9>] ? kthread+0x79/0x81
[  857.210424]  [<ffffffff81003694>] ? kernel_thread_helper+0x4/0x10
[  857.211141]  [<ffffffff8104fc80>] ? kthread+0x0/0x81
[  857.211791]  [<ffffffff81003690>] ? kernel_thread_helper+0x0/0x10
[  857.212497] ---[ end trace e194c9c11700625f ]---
[  857.265004] alua: device handler registered
[  857.290275] device-mapper: multipath round-robin: version 1.0.0 loaded
[  857.291599] sd 0:0:0:0: alua: supports implicit and explicit TPGS
[  857.292794] sd 0:0:0:0: alua: port group 00 rel port 02
[  857.293645] sd 0:0:0:0: alua: port group 00 state A supports TOUSNA
[  857.300249] sd 0:0:0:0: alua: port group 00 state A supports TOUSNA

The Open-FCoE tcmfc target should look like this:

[  491.470139] host0: Assigned Port ID  10102
[  491.470789] host0: libfc: Port ( 10102) entered point-to-point mode
[  491.471679] TARGET_CORE[fc]: Registered fabric_sess_ptr: ffff88007d474340

Notice that Port ID 10101 is being used for the FCoE Initiator, and Port ID 10102 used by the FCoE Target.

lsscsi

The lsscsi output is as follows:

lenny64guest0:~# lsscsi
[0:0:0:0]    disk    LIO-ORG  FILEIO           3.1   /dev/sda
lenny64guest0:~# lsscsi --transport
[0:0:0:0]    disk    fc:0x200000e081c090b3,0x010102  /dev/sda

sg_inq

The sg_inq -i output for the EVPD 0x83 identifier is as follows.

lenny64guest0:~# sg_inq -i /dev/sda
VPD INQUIRY: Device Identification page
 Designation descriptor number 1, descriptor length: 20
   designator_type: NAA,  code_set: Binary
   associated with the addressed logical unit
     NAA 6, IEEE Company_id: 0x1405
     Vendor Specific Identifier: 0xf776de20d
     Vendor Specific Identifier Extension: 0xc416d4382d8b16d0
     [0x6001405f776de20dc416d4382d8b16d0]
 Designation descriptor number 2, descriptor length: 56
   designator_type: T10 vendor identification,  code_set: ASCII
   associated with the addressed logical unit
     vendor id: LIO-ORG
     vendor specific: FILEIO:f776de20-c416-4382-8b16-0519a7dcf3ab
 Designation descriptor number 3, descriptor length: 8
   transport: Fibre Channel (FCP-2)
   designator_type: Relative target port,  code_set: Binary
   associated with the target port
     Relative target port: 0x2
 Designation descriptor number 4, descriptor length: 8
   transport: Fibre Channel (FCP-2)
   designator_type: Target port group,  code_set: Binary
   associated with the target port
     Target port group: 0x2
 Designation descriptor number 5, descriptor length: 8
   designator_type: Logical unit group,  code_set: Binary
   associated with the addressed logical unit
     Logical unit group: 0x0
 Designation descriptor number 6, descriptor length: 40
   transport: Fibre Channel (FCP-2)
   designator_type: SCSI name string,  code_set: UTF-8
   associated with the target port
     SCSI name string:
     20:00:00:e0:81:c0:90:b3,t,0x0001

sg_persist

The sg_persist -c (Persistent Reservations) output:

lenny64guest0:~# sg_persist -c /dev/sda
 LIO-ORG  FILEIO  3.1
 Peripheral device type: disk
Report capabilities response:
 Compatible Reservation Handling(CRH): 1
 Specify Initiator Ports Capable(SIP_C): 1
 All Target Ports Capable(ATP_C): 1
 Persist Through Power Loss Capable(PTPL_C): 1
 Type Mask Valid(TMV): 1
 Allow Commands: 1
 Persist Through Power Loss Active(PTPL_A): 0
   Support indicated in Type mask:
     Write Exclusive, all registrants: 1
     Exclusive Access, registrants only: 1
     Write Exclusive, registrants only: 1
     Exclusive Access: 1
     Write Exclusive: 1
     Exclusive Access, all registrants: 1

sg_rtpg

The sg_rtpg output (ALUA) output:

lenny64guest0:~# sg_rtpg /dev/sda 
Report target port groups:
 target port group id : 0x0 , Pref=0
   target port group asymmetric access state : 0x00
   T_SUP : 1, O_SUP : 1, U_SUP : 1, S_SUP : 1, AN_SUP : 1, AO_SUP : 1
   status code : 0x00
   vendor unique status : 0x00
   target port count : 00
 target port group id : 0x1 , Pref=0
   target port group asymmetric access state : 0x00
   T_SUP : 1, O_SUP : 1, U_SUP : 1, S_SUP : 1, AN_SUP : 1, AO_SUP : 1
   status code : 0x00
   vendor unique status : 0x00
   target port count : 01
   Relative target port ids:
     0x01
 target port group id : 0x2 , Pref=0
   target port group asymmetric access state : 0x00
   T_SUP : 1, O_SUP : 1, U_SUP : 1, S_SUP : 1, AN_SUP : 1, AO_SUP : 1
   status code : 0x00
   vendor unique status : 0x00
   target port count : 01
   Relative target port ids:
     0x02

Timeline

History of LIO Target
Release	Details	2009												2010												2011												2012												2013
Release	Details	1	2	3	4	5	6	7	8	9	10	11	12	1	2	3	4	5	6	7	8	9	10	11	12	1	2	3	4	5	6	7	8	9	10	11	12	1	2	3	4	5	6	7	8	9	10	11	12	1	2	3	4	5	6	7	8	9	10	11	12
4.x	Version																								4.0
	Feature																								Target Core			Loop back			FCoE			iSCSI		Perf		SRP			CM WQ		FC USB 1394			vHost		Perf		Misc		16 GFC		iSER			NTB
	Linux																								2.6.38			2.6.39			3.0			3.1		3.2		3.3			3.4		3.5			3.6		3.7		3.8		3.9		3.10			3.11
3.x	Version	3.0								3.1		3.2						3.4					3.5
3.x	Feature	ConfigFS Loopback								PRs ALUA		Cleanups						Open-FCoE					Backports

Acknowledgements

FCoE was made possible by Joe Eykholt's generous contributions to tcm_fc, and his ongoing work of making the Linux libfc target capable.

Notes

↑ Kiran Patil (5/17/2011). "[SCSI] tcm_fc: Adding FC_FC4 provider (tcm_fc) for FCoE target (TCM - target core) support". lkml.org.

External links

FibreChannel (FC) Wikipedia entry
FibreChannel over Ethernet (FCoE) Wikipedia entry
Open-FCoE

[0] Kiran Patil (5/17/2011). "[SCSI] tcm_fc: Adding FC_FC4 provider (tcm_fc) for FCoE target (TCM - target core) support". lkml.org.

[1]

Fibre Channel over Ethernet

The Linux SCSI Target Wiki

Contents

Specifications

Setup

Initiator

lsscsi

sg_inq

sg_persist

sg_rtpg

Timeline

Acknowledgements

See also

Notes

External links

Personal tools

Namespaces

Variants

Views

Actions

Search

Navigation

Toolbox

Google AdSense


Cisco Systems FCoE fabric module
Original author(s)	Joe Eykholt Kiran Patil
Developer(s)	Cisco Systems, Inc.
Initial release	July 21, 2011 (2011-07-21)
Stable release	4.1.0 / June 20, 2012; 10 months ago (2012-06-20)
Preview release	4.2.0-rc5 / June 28, 2012; 10 months ago (2012-06-28)
Development status	Production
Written in	C
Operating system	Linux
Type	Fabric module
License	GNU General Public License
Website	www.cisco.com