Introduction to Fiber Channel Switching
Amr Ibrahim Enan is a Global Knowledge instructor who teaches and blogs from Global Knowledge Egypt.
With the introduction of I/O consolidation that I discussed in a previous post, network engineers found themselves responsible for one physical switch that provides both LAN and SAN access for servers in data centers since SAN traffic is now carried over Ethernet using FCOE technology.
Even if you do not have to configure the SAN features of your I/O consolidation switch, you should at least have a basic understanding of FC technologies. In this post, I will provide you with all you need to know about FC technologies as a network engineer. While doing so, I will try to relate — whenever possible — the FC feature to a network feature you already understand, which will make it much easier and fun to understand and memorize the relative new FC feature.
Before we discuss anything, let me begin with a quick comparison between the TCP/IP stack and the FC stack.
Before we dig deeply in this comparison, I just want to bring to your attention that for the TCP/IP stack, we have four layers. Each layer has a programming API or physical interfaces, but for FC it is called Levels. Theoretically speaking there is no difference, but from the implementation perspective Levels does mandate programming API or Physical interfaces between levels where levels implementation is vendor specific. So how this will makes FC different from Ethernet? In Ethernet, if you connected two Ethernet switches in which one is Cisco and one is Juniper, most probably it will work just fine as each switch functionality is strictly defined and implemented according to its layer definition. For FC switches that is not the case if you tried to connect a Cisco Switch to a Brocade switch for example. Because of this specific implementation, you have to configure both switches in what is called compatibility mode since each vendor has a specific FC level implementation.
Now let us dip deeper in the layers. As you can see in Figure 1, we have the TCP/IP stack with its four layers, and right next to it is the FC levels .
Application layer/FC-level 4 and FC-level 3
The application layer is where we run all of our application layer protocols like DNS SMTP,HTTP, etc. As you may already know, whenever any application layer protocol needs to put some data on the network, it relies on the services of the next layer which, in our case, is the Transport layer. This is exactly the same role of FC level 4. In this layer, we map the ULP protocols to the FC environment.
You can see from the above figure that inside FC-level 4 each ULP has it is own mapping. FC level 3 is a placeholder for future features that can be applied to all ULP protocols before the packets is passed to FC level 2
Transport layer /FC level 2
In the transport layer we have two main protocols: the TCP protocol and the UPD protocol. Both are available for the use of application layer protocols, but each provides a different set of services to the application layer. TCP is a connection oriented protocol, which means it needs to negotiate a specific set of parameters before exchanging any data between the two peers on the network. Thanks to this negotiation, TCP can recover lost frames while sending data and in times of congestion, provide the connection with flow control service. UDP is just a connectionless protocol that does not provide you with any of those services. FC level 2 provides FC level 4 with the same set of services that are provided by TCP to the Application layer while TCP is implemented inside the kernel of the OS. This places a lot of processing load on the CPU to drive a TCP connection. For example, to drive a 100 mb/s link TCP would consume 80% of your CPU cycles. FC is implemented in Hardware using smart Asics on the Storage adapter, meaning that FC level 2 has its own CPU instead of relying on the system CPU This is why storage adapters named HBA (Host bus adapters ) are more expensive than normal network adapters and require virtually no system CPU cycles to drive the FC link.
Again, the transport layer will rely on the Internetwork layer to put this data on the network. In the Internetwork layer, we have only one protocol called the IP protocol. The main function of the IP protocol is to provide a unique identity for your machine on the network so it can send and receive data from other hosts connected to the Internet. This address is used by routers to identify the path to this host through routes exchanged by the routers using any configuration routing protocol. This layer has no equal level in FC.
Data link /FC level 1
Finally, the Internetwork layer relies on the Data link layer too, but this piece of data on the network is actually two layers. The Data link layer is where we have the Ethernet protocol encoding that decodes the data as it moves to and from the host. This function is carried by FC level 1. For FC we use 10b/8b decoding scheme. Because of this decoding scheme, you will end up losing 20% of your BW. This means that if you use 1Gb/s, your effective BW is 8oo MB/s. How do we calculate it? It is very easy. Just multiply 1 G/s* 8/10, and the result is 0.8 G/s, which is the same as 800 MB/ s.
Physical layer / FC level 0
Both layers are responsible for serializing the data to the cable and deserializing the data off the cable. Take notice that FC can provide you with an error free line if the latency is no higher than 10^−13 .
In the next post we will the difference between Ethernet switching and FC switching concept
Ethernet Switching vs. Fiber Channel Switching
AUTHOR: AMR ENAN 6 JUNE 2012 15,454 VIEWS 3 COMMENTS
TAGS: ETHERNET SWITCHING, FIBER CHANNEL SWITCHING
TAGS: ETHERNET SWITCHING, FIBER CHANNEL SWITCHING
Amr Ibrahim Enan is a Global Knowledge instructor who teaches and blogs from Global Knowledge Egypt.
Now that we discussed the difference between the TCP/IP stack and the FC levels, it is time to examine the difference between Ethernet and FC switching concept
But first, let me ask you what is the major difference between Ethernet switching and FC switching? Ethernet is loose, which means if a device sends a frame to an Ethernet switch and the receiving buffer of that switch is full, the switch silently drops the frame. It will not notify the sending host that it dropped the frame, so the question is how we can recover this frame?
Well, actually Ethernet does not care about that. It leaves this job to the other layer in the TCP/IP stack: the Transport layer. If your application layer protocol relies on TCP to transport the data, TCP will detect the loss of the segment, and it will recover the segment by asking the sending host to resend this segment again. Also, TCP has a flow control mechanism. If the receiving host buffer becomes full then it can ask the sending host to stop sending frames for a while. This can be simply done by sending a TCP segment after setting the window size to zero. By doing so, I am asking the sending host to stop sending me any more segments until I notify it that I can receive more segments.
Believe it or not, we have a similar feature in Ethernet called Flow control, but it is disabled by default on Ethernet switches. If you enabled this feature on an Ethernet switch and the receiving switch dropped a frame, it will send a pause frame to the sending switch asking it to stop sending frames for a specific amount of time. That’s cool, is it not? But no one uses this feature since TCP handles this job just great for ip traffic.
Unlike Ethernet, FC is LOSSLESS, which means that there is no chance that a frame will be dropped when it is transmitted from a host to a switch or from a switch to another switch. This is why back in the days when they decided which network technology was better to handle SCSI traffic between the hosts and centralized storage, FC was the only choice at that time. But wait a minute, what is SCSI protocol? Oh, sorry, I forgot you’re the network guys.
SCSI stands for Small Computer System Interface, and it is the standard protocol used to carry the data between the OS and the hard disk. SCSI used to be the definition of the protocol and the cable carrying this protocol, but as the need grew for more centralized storage, the SCSI cable was not fit for such a task for many reasons. We had two technologies at that time that satisfied our needs, which were:
§ All servers should have access to the storage to efficiently utilize and manage it
§ When I upgrade storage capacity, I do not have to shut down the servers introducing down time
Both technologies, Ethernet and FC, satisfied the two conditions at that time, but there was a catch. The SCSI protocol, when it was first designed, was meant to travel over very small distances. This means that it was very unlikely that we lost any SCSI blocks over such a small distance, which is why, in SCSI protocol, there is not much stress on frame recover mechanisms
If SCSI lost a frame, it required one minute to recover that lost frame. ONE MINUTE! Yes, you read it right. A COMPLETE 60 seconds to recover one SCSI data unit. This means if we decided to send SCSI over the network, then the network protocol we use should be LOSSLESS. This is why, at that time, FC was the only choice.
But how does FC achieve this lossless behavior? Well, FC is completely different from Ethernet. In Ethernet, once a host is connected to a switch or a switch is connected to another switch, they can start exchanging data immediately with no prior negotiations whatsoever. But in FC, it is different when you connect any device to your FC Switch. You should indicate to your FC switch what the type of the device is on the other end. Is it a switch, a host, or a storage array, because based on the type of the device on the other end, the switch will start a specific set of negotiations to ensure that while exchanging data with the device on the other end, no frames will be lost during data exchange.
So how we can indicate to the switch what type of device is on the other end, and what is the set of negotiations between a switch and another switch or between a switch and a host?
This is what we will discuss in the next post.
FLOGI, PLOGI, and PRLI
AUTHOR: AMR ENAN 27 JUNE 2012 10,831 VIEWS 6 COMMENTS
TAGS: FIBER CHANNEL SWITCHING
TAGS: FIBER CHANNEL SWITCHING
Amr Ibrahim Enan is a Global Knowledge instructor who teaches and blogs from Global Knowledge Egypt.
When you connect your server or storage, it will send a Fabric login (FLOGI) request. This FLOGI request serves two purposes:
§ Fibre Channel ID is the WWPN connection to this port from the FLOGI server
§ Exchange buffer credits with the switch
To better understand the first function, it is exactly the same if I rewrite it as, “request an IP address for a MAC address from the DHCP server”.
Yes, that’s right, FCID in the FC world is more or less the same as an IP address in the networking world. An IP address has two main IDs. The network ID is unique to devices that exist in the same Layer 2 domain, and the host ID is unique to each host.
FCID has two main parts: the Domain ID is unique for all devices that are connected to the same switch, and the Port ID is unique to each device connected to the same switch.
If you connect your laptop to your network, what is the first thing it will do? It will request an IP address so that it can communicate with other devices. In order to do so, it will send a DHCP BC frame that contains the MAC address. The equivalent to MAC in the FC world is WWPN, and the equivalent to a DHCP server is the LOGIN server (a service running inside the FC switch).
LOGIN server is a service running inside each storage switch. It is responsible for providing a unique FCID to each device connected to your switch, just like the DHCP provides each network host with a unique IP address.
FLOGI is functional in that it provides the buffer credit exchange between server and switch.
The FC is lossless, and the way it achieves this is on a link-by-link basis, meaning that the two devices on the same link will exchange buffer credits with each other, but why?
Checking the above figure, you’ll see that the TX (Server) now knows that the maximum number of frames RX (Switch) can hold in the buffer is 4. It will then send another variable called BB_Credit_CNT. At the login, it will send its value to 0 and whenever the TX sends a frame to the RX, the value will increase by 1. Then TX can send frames to RX until the value of BB_Credit_CNT is less than the BB_Credit. When a buffer space is offloaded on the RX side, an upper layer RX will send the TX a RDY frame. When the TX receives that frame it will decrease the value of BB_Credit_CNT by one and be able to send one more frame to TX.
All of this is what makes FC lossless. No frame will be sent from one device to another unless it was a switch to switch or server to switch.
Port Login (PLOGI)
After each device finishes the FLOGI phase, they now each have their own FCID. But they don’t know how to reach other devices connected to the same switch or connected to other switches in the fabric. In order to communicate with other devices, each device will need to do PLOGI.
There’s another service running inside the switch called Name server. When any device registers with this server, it sends its WWPN map to its FCID. The Name server will expose that device and all the register requests it receives this way can communicate with each other.
And Finally Process Login (PRLI)
As the two devices can now communicate with each other using the FC network SCSI, communication now takes place between the two devices. The SCSI has two main operations which are SCSI read or SCSI write.
In my next post, I’ll demonstrate a communication scenario between server and storage and explain some advanced configuration options that network engineers will be interested in.
Fiber Channel Switching in Deep
AUTHOR: AMR ENAN 20 JUNE 2012 5,479 VIEWS ONE COMMENT
TAGS: ETHERNET SWITCHING, FIBER CHANNEL SWITCHING
TAGS: ETHERNET SWITCHING, FIBER CHANNEL SWITCHING
Amr Ibrahim Enan is a Global Knowledge instructor who teaches and blogs from Global Knowledge Egypt.
Previously I gave you a brief overview on the differences between Fiber Channel switching and Ethernet switching. The major difference between FC and Ethernet is that FC is lossless while Ethernet is lossly. In this post, I’ll explain how FC achieves a lossless behavior.
First, let me ask you what is the difference between TCP and UDP protocol? TCP provides you with extra services that UDP doesn’t, like lost frame recover flow control. But, you might ask, how? Before any two hosts communicate using TCP, they will first go through some sort of negotiation on parameters for those services to be possible. And that is exactly what happens in FC switching.
Whenever you connect a storage device or a server to a storage switch, before they exchange any data frames, they will do some negotiations to guarantee lossless behavior. Also, when you connect a FC switch to another FC switch they will perform some sort of negotiating to guarantee lossless behavior. But, negotiations that the switch does with one switch are completely different from the negotiations that the switch does with another switch.
The port type needs to be configured on FC switches but not on Ethernet switches. You’ll need to tell the switch what type of device is on the other end so it knows which set of negotiations it should use (a switch-to-switch negotiation or a server/storage-to-switch negotiation).
In the above figure, you can see that when you connect a switch with a server you need to configure the switch port as F_port, and if you’re connecting a storage device to a switch you should configure it as FL_port. If you are connecting a switch to another switch, then you should configure it as an E_port. Now let’s examine the negotiation that takes place between the server and the switch.
One thing that makes storage communication easy to understand is that we only have one protocol we need to understand in FClevel 4, which is SCSI protocol. SCSI has two main operations, SCSI read and SCSI write, which are always initiated from the server to the storage.
As you can see in the figure above, before the server can send a data frame to storage both devices need to go through three negotiations phases. Two phases are done between the server and the switch:
§ FLOGI (Fabric login )
§ PLOGI(Port login )
The final phase is done during the SCSI process, running on both devices:
§ PRLI (Process login )
In the next post, we’ll talk about a FLOGI and PLOGI requests and the purposes it serves.
Market players:
There are four main players in the market: Cisco, Brocade, Qlogic, Emulex. Cisco entered this market by Andiamo acquisition by delivering MDS 9000 series platforms.
- http://blog.globalknowledge.com/technology/cisco/routing-switching/introduction-to-fc/
