Error Recovery Level
The Linux SCSI Target Wiki
+ / \ / 2 \ <-- Connection recovery +-----+ / 1 \ <-- Digest failure recovery +---------+ / 0 \ <-- Session failure recovery +-------------+
The following table provides an overview over the error recovery capabilities expected from each error recovery level implementation respectively.
|ERL||Associated Error Recovery Capabilities|
|0|| Session recovery class|
(Section 126.96.36.199 - Session Recovery)
|1||Digest failure recovery (see note below) plus ERL=0|
|2|| Connection recovery class|
(Section 188.8.131.52 - Connection Recovery) plus ERL=1
Note: Digest failure recovery is comprised of two recovery classes: Within-Connection recovery class (Section 184.108.40.206 Recovery Within-connection) and Within-Command recovery class (RFC 3270 Section 220.127.116.11 "Recovery Within-command").
The following sections provide more detail over the error recovery capabilities expected from each error recovery level implementation.
ERL=0: Session Recovery
ERL=0 (Session Recovery, iscsi_target_erl0.c) is triggered when failures within a command, within a connection, and/or within TCP occur. This causes all of the previous connections from the failed session to be restarted on a new session by sending a iSCSI Login Request with a zero TSIH.
Restart all iSCSI connections on any failure.
This is a special case for ERL=0 and recovering the existing I_T Nexus.
ERL=1: Digest Failure Recovery
ERL=1 (Digest Failure Recovery, iscsi_target_erl1.c) only applies to traditional iSCSI. For iSCSI/SCTP (which has its own CRC32C) and both types of iSER (so far), handling header and data checksum recovery can be disabled.
Within Connection Recovery
- CmdSN Retry Timer
- Logic to handle Recovery R2T
Within Command Recovery
- DataOut Timer
- Datain Timer
Logic to handle recovery / generate R2Ts
Logic to issue SNACK for missing StatSN or DataIN
- Support RDATA SNACK [this is still TODO]
ERL=2: Connection Recovery
ERL=2 (Connection Recovery, iscsi_target_erl2.c), also known as "Internexus-MP", is an optional RFC-3720 feature that allows for both single and multiple communication path sessions within a iSCSI I_T Nexus (and hence the SCSI Nexus) to actively perform realligence/retry on iSCSI ITTs from failed iSCSI connections. ERL=2 allows iSCSI fabrics to take advantage of recovery in all regards of transport level fabric failures, and in a completely OS independent fashion (i.e. below the host OS storage stack).
As being a SCSI feature, it is generic to the underlying network protocol (fabric module), and has been implemented with iSCSI/TCP, iSCSI/SCTP, and is possible for iSER/DDP, iSER/IB. With the latter case (iSER) traditional iSCSI recovery logic that pertains to ERL=1 is disabled, as the underlying RCaP is handling integrity of payloads using CRC32C or greater checking.
Note that in ERL=0 sessions, all communication paths need to be shutdown/restarted after a recovery exception occurs.
- Handle Logout Request
- REMOVECONNFORRECOVERY (CSM-E)
- Handle generation of Recovery R2Ts for WRITE
- Traditional iSCSI, iSER
- Handle recovery DATAIN for READ
- Traditional iSCSI, iSER
- Handle changed MaxRecvDataSegmentLength across ERL=2
- Traditional iSCSI [TODO], iSER [TODO]
Handle new Login Request for existing iSCSI connection handle (CSM-I).
- RFC 3720: Internet Small Computer Systems Interface (iSCSI)
- RFC 5046: Internet Small Computer Systems Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)
- SCSI: Persistent Reservations (PR), Asymmetric Logical Unit Assignment (ALUA), Multiple Connections per Session (MC/S)
- Fabric modules: Fibre Channel, FCoE, iSCSI, InfiniBand and vHost
- Management: targetcli
- ↑ "Internet Small Computer Systems Interface (iSCSI)". RFC-3270. April 2004.