Should I be using Fiber Channel, iSCSI or FCoE?

Question: We are bringing our first SAN into work and we have been presented with three options for connectivity.  Which one should we be using, Fiber Channel (FC), iSCSI or Fiber Channel over Ethernet (FCoE)?

Answer: This is really an it depends question.  Some of the things that the decision needs to be based on include:

  • The amount of data that you'll be moving between the servers and the storage array.
  • Your budget
  • The number of free 1 Gig and 10 Gig ports available in your network infrastructure.
  • Your experience with zoning
  • What technologies does your tape backup solution support
  • Will virtual machines need direct access to the storage array
  • How many servers will be accessing the storage array

Some of the big benefits of these technologies also tie directly to their down sides.  Let's review these three technologies one by one.

Fiber Channel

Fiber channel has been around for decades as the most well known, well baked storage solution available.  Fiber channel is very quick, with a very low latency with moving data around the fiber channel network.  Port speeds are increasing from 8 Gigs per port to 16 Gigs per port with Emulex's announcement in November when they announced their 16 Gig HBAs (Host Buss Adapters) and with Brocade bringing out their 16 Gig Fiber Channel switches. 

Related: My SAN admin wants to put my transaction logs on FAST storage. Should we?

Ignoring the 16 Gig ports for a minute as they are basically so new that you won't have them in your shop for months or years, lets look at the 8 Gig and below systems for a minute.  When you buy a fiber channel switch these days it'll support basically anything 8 Gigs / second or slower.  So if you've got some machines that are older and that old support 4 Gig HBAs or even 2 Gig HBAs those will work just fine as you can simply put 2 Gig or 4 Gig SFPs into the switch (the little connectors that let the fiber cables connect to the switch).

Many of the newer tape backup solutions out there only support fiber channel connections at this point for backing up directly from the storage array.

Related: Can the fiber channel switch be a bottleneck?

Now there are a few downsides to fiber channel.  These revolve around the need to properly setup zoning within the fiber channel switches, granted this isn't actually all that hard, but a simple mistake can stop a lot of servers from talking to the storage array.  A couple hours of training and you'll know all you need to know about how to do this.  I'd recommend getting a little training on it instead of having a consultant setup the zoning so that you know how to make the changes yourself in the future.  Another is the cost of the switches.  Fiber channel switches cost a lot more per port than Ethernet switches do, which means that the cost of getting into fiber channel is going to be quite a bit higher.  For example I just purchased a couple of Fiber Channel switches (Cisco 9124s) which are 24 port switches (which come with only 8 ports per switch active) and they cost ~$4300 each plus support.  That's about $530 per port.  Now when I go to activate the rest of the ports they will cost less than that as I already have the hardware, but that's a pretty expensive cost per port.

iSCSI

iSCSI was introduced several years ago, not long after 1 Gig Ethernet ports started becoming pretty standard on servers.  Back then if you had asked me if iSCSI was OK to use on a production network my answer was pretty simple; "No".  Today that answer has changed from "No" to "probably".  If you've got 10 Gig NICs and 10 Gig Switch Ports and 10 Gig ports on the storage array then iSCSI should work just fine, provided that you have the iSCSI network isolated from the rest of the network on it's own vLAN or separate network switches.  For iSCSI designs where the storage IO requirements aren't all that high sharing the network switches with the rest of the network is fine, which can be a huge cost savings.  However when you'll be moving a lot of data over the iSCSI network I always recommend moving the servers and the storage over to their own network switches to reduce the latency and contention on the network.

There are a few downsides to iSCSI however.  The first being that it runs over TCP/IP which means that it's got the same timeouts and retries as everything else which runs over TCP/IP.  While these timeouts and latencies work fine for file servers and web browsing, they aren't always the greatest for when moving storage around.  The second is the ability to overload the Ethernet network causing end user performance problems.  The third revolves around security.  If you have which is sensitive and you are storing that data in plain text within the database, someone could capture the network packets of the iSCSI communications between the server and the storage array and use that to view your customer data.  iSCSI can easily run over 1 Gig or 10 Gig ports, but it'll probably be a while before faster ports are available which means that you'll need to aggregate 10 Gig ports together to get more bandwidth.  There is also the annoyance that a decent percentage of the TCP packets are header information.

FCoE

Fiber Channel over Ethernet is the newest of the three storage communication technologies.  It uses traditional Fiber Channel communication but using Ethernet instead of fiber cables to transfer the data.  You can find some interesting reading about the standard and how it merges Fiber Channel and Ethernet here.  Unfortunately not a lot of venders support FCoE at the moment.  Cisco and Brocade both have switches which support FCoE (Cisco was one of the companies who started the build FCoE) but only a few server venders have FCoE cards (regular NICs aren't supported) and only a couple of storage venders support FCoE on their storage platforms.  Another problem is that the FCoE spec isn't finished yet.  A new version of the spec was supposed to be released Q3 or Q4 of 2011, but that didn't happen so some features of the FCoE protocol aren't supported yet.  And since the new spec won't be ready until at least Q2 or Q2 of 2012 at this point it'll be a bit until these features are supported.

A problem that I have run across is that multihop isn't really supported in FCoE, yet.  The basic jist of this multihop problem is that your FCoE traffic can't pass from switch to switch without going multihop and FCoE doesn't like multihop because at its core FCoE is actually fiber channel and has to follow all the rules of Fiber Channel.  J Metz (blog | blog | Twitter) has a great blog post talking about what Multihop is and how all this works in a much more technical way than I possible could.

Based on all this information, my preference today is to stick with good old Fiber Channel.  In fact the data center which I'm currently building for Phreesia was going to be using FCoE but with the missing features multihop issue with FCoE we had to switch back to Fiber channel. With the 16 Gig announcements that have been coming out, I'm glad that we did.

Hopefully this helps you make your decision when selecting the storage technology for your data center.

Denny

Discuss this Blog Entry 15

on Dec 14, 2011
Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz
on Dec 14, 2011
Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz
on Dec 14, 2011
Note: This comment is being posted on behalf of J. Metz of Cisco who tried to post however ran into issues with this sites commenting mechanism. Since my comment posted he asked if I would post the following on his behalf. Note that the following is via J and do not reflect my opinions/recommendations nor should be seen as an endorsement. +++ On Denny's blog: Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz jmmetz@cisco.com twitter @jmichelmetz ++++ Direct comments to J at above contacts... +++
on Dec 14, 2011
Disclosure: This comment is being posted on behalf of J. Metz of Cisco who tried to post however ran into issues with this sites commenting mechanism. Since my comment posted he asked if I would post the following on his behalf. Note that the following is via J and do not reflect my opinions/recommendations nor should be seen as an endorsement. <<<< On Denny's blog: Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz=jmmetz@cisco.com twitter @jmichelmetz >>>> Please direct any comments to J via above contacts... Cheers gs
on Dec 14, 2011
Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz
on Dec 14, 2011
Denny good point about "it depends" which to use. However there is another option which is not commonly talked about when it comes to setting up a shared storage environment aka SAN which is shared SAS, either switched or non-switched. Here is a link ( http://storageioblog.com/?p=2044 ) to learn more about shared SAS, the other SAN. The pro of shared SAS is that many server vendors (Cisco, Dell, HP, IBM, Oracle, etc) are providing native SAS including 6Gb support on their servers similar to native Ethernet along with support for PCIe SAS adapters and blade server mezzanine cards. There are also storage systems that support a mix of native iSCSI, SAS and FC (some with FCoE support or ready for it) including from Dothill, Dell, HP, IBM, NetApp, Oracle, Promise etc. To increase the number of usable ports on those storage systems from the usual four or eight SAS ports, SAS switches can be used as is the case with some vendors prepackaged systems. Note that for VMware environments there is even HCL support. Benefits include relative low cost similar to 1Gb E iSCSI with performance closer to 8Gb FC. Caveats include limited distance of a few meters (e.g. inside a cabinet or adjacent cabinet), not as widely discussed as not as many people read, hear or know about the capability. Compared to iSCSI or FC/FCoE, server attached connectivity will be limited in terms of 10s of servers vs. potential for 100s. Of course, the correct answer of when to use iSCSI, SAS, FC, and FCoE for shared block based storage access (e.g. SAN), each of which have their pros/cons including when/where to use or not use is "it depends"... They are all good when used where applicable along with where the strengths exist. Cheers
on Dec 14, 2011
Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz
on Dec 14, 2011
Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz
on Dec 14, 2011
Hey thanks - its a good read.
on Dec 14, 2011
Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz
on Dec 14, 2011
Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz
on Dec 14, 2011
Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz
on Dec 17, 2011
Well, I'm a big fan of iSCSI and the technology is getting better all the time. You cite security and performance as in efficiencies inside iSCSI, but that simply doesn't hold up in my opinion. CHAP authentication and IPSec are ways of ensuring the security and also segregating to multiple private network paths for performance and security. The latest iSCSI initiiators have an extensive policy set to provide failover and round robin activity too. The beauty of iSCSI is that you need run of the mill network equipment such as NICs, switches,etc and TCP\IP setup skills. They each have their pros and cons regards Perry Whittle
on Dec 14, 2011
Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz
on Dec 14, 2011
Hello Denny, Thank you for putting this together. Most of the information you provide is very useful. However, there are a few items regarding FCoE that need to be corrected clarified. First and most important, the FCoE "spec" has been finished and finalized for a long time. The T11 vote for FC-BB-5 (the standard document for FCoE, including Multihop FCoE) passed the final vote in 2009, long after it was technically stabilized in 2007. INCITS published the standard in May, 2010. Like any standards document, additional standards are developed that includes new features, much like operating systems add features with new revisions. This is why the standards bodies continue to work on new document revisions, not because earlier ones weren't finished. Second, inasmuch as you state that there are not a lot of vendors, I'm afraid your information must be out of date. There are dozens of companies that have FCoE-related equipment available now. Plugfests at the UNH Interoperability Labs recently conducted its seventh FCoE plugfest, which included over a dozen companies. Easily a dozen more have FCoE equipment (i.e., CNAs, switches, storage arrays). VMWare has upped the ante with considerable FCoE support in VSphere 5. Every major server manufacturer - HP, Dell, IBM, Cisco, e.g. - have CNAs available for their product lines. (Disclosure, I work as a Product Manager for FCoE for Cisco, so granted my information may be a bit more readily available). Hope this helps. Best, J Metz

Please or Register to post comments.

What's Troubleshooting SQL Server Storage Problems?

Practical advice, insight, and help for core SQL Server considerations.

Contributors

Denny Cherry

Denny Cherry is the owner and principal consultant for Denny Cherry & Associates Consulting and has over a decade of experience working with platforms such as Microsoft SQL Server, Hyper-V,...
Blog Archive

Sponsored Introduction Continue on to (or wait seconds) ×