HP just published an advisory describing how to tune some parameters for Emulex, Qlogic and Brocade Fibre Channel HBAs: c02518189. It sounds like these are new, but these changes have been around for at least 6 months in all three vendors’ HBAs.
“Emulex driver version 2.42.002 or later, along with OneCommand Manager version 184.108.40.206 or later,”
Use HBAnywhere to change these.
Examples to tune the server or port level transfer size:
- 128 kbytes, set the LimTransferSize = 2 and ExtTransferSize = 0 (default)
- 512 kbytes, set the LimTransferSize = 0 (default) and ExtTransferSize = 0 (default)
- 1 Mbytes, set the LimTransferSize = 0 (default) and ExtTransferSize = 1
This is part of the Qlogic SANSurfer utility.
- c:\>qlfc -tsize /fc
- c:\>qlfc -tsize /fc /set 128
- c:\>qlfc -tsize /fc /set default
- bcu drvconf –key bfa_max_xfer_len –val 64
- bcu drvconf –key bfa_max_xfer_len –val 128
Just took the BCFD (brocade certified fabric designer) exam two days ago.
Bring some water and food.
Good exam, but I am really tired of exams and certifications for now!
Also, isn’t easy to confuse Brocade Certified Fabric Designer with somebody who makes clothes?
Check out my other posts on the BCFD subject:
- Given a scenario, design a solution that meets the customer’s requirements
- Given a scenario, demonstrate knowledge of resiliency, redundancy, HA, and locality
- Given a scenario, describe the various documents required in the design assessment
Practice by making up many scenarios and then deciding which is the best way to design it.
Management and Monitoring Tools
- Given a scenario, describe how to satisfy a specific monitoring requirement
- Demonstrate knowledge of Brocade management tools
What to monitor
How to monitor these
Hardware/Software Products and Features
- Demonstrate knowledge of interoperability of B-Series/M-Series products
- Given a scenario, describe Brocade hardware products and their purpose
- Given a scenario, demonstrate knowledge of Brocade software features and purpose
Features: VF, FCR, TI, QoS, FW, IRL, Trunking, Port Fencing, D_Port
- Given availability, performance and distance requirements, design an appropriate long distance solution using Fibre Channel
- Given a specific set of requirements, demonstrate ability to design a SAN extension solution using FCIP
FastWrite, Tape Pipelining, SACK
Max distance for LWL and ELWL:
Max performance of FCIP:
Performance Tuning Optimization
- Given a performance scenario, determine an appropriate solution
- Describe strategies for maximizing throughput in a Data Center Fabric
ICL, nohops, trunking.
How to increase performance in FCIP and FCR:
Migration and Integration
- Given an existing fabric, identify migration strategies to upgrade the fabric with new technology
- Given a set of existing fabrics and network devices, determine a consolidation plan that minimizes disruption
- Describe the requirements to integrate a Brocade DCX Backbone into an existing M-Series fabric
FCR, Integrated Routing, E_port on a switch in the M_series.
- Identify requirements for restricting which switches/devices may join a fabric
- Identify security features to restrict administrative access to a switch
This is a post in series of me studying for the BCFD – Brocade Certified Fabric Designer and it’s my comments on the document SAN Design Best Practices. Apparently this document is planned to be updated. The one I have is version 2.1. To find the latest go to My.brocade.com , documentation, Best Practices Guides. There’s also a “SAN Migration” guide there, but it’s from 2003 so irrelevant when it comes to anything specific, but ideas and reasons and methods might be valuable.
OK. I thought about doing something similar for this document as for the previous ones. But I just don’t feel like that, it’s basically just re-writing things in different wording so that it sticks in my brain. No instead I’ll post the questions that popped into my brain while reading it.
For a starter, I printed this .pdf. OK it’s not so environmentally friendly but it’s nice to have a break. One thing though, it took me a lot longer to read this than the course modules for BCFD. The SAN Design Best Practices is a first class pdf. At least in my opinion. I mean it’s general and specific. It needs to be general because there’s a lot of reasons behind designing things. Also, I don’t have any actual previous experience designing a SAN, so this is all new to me, and brings up a new side of Storage and Storage Networking that I just haven’t bothered much with before. Hopefully I have and will be learning a lot.
This paper refers to a lot of documents.
The “Brocade Scalability Guidelines” is not updated with 16G products (Only goes to FOS 6.3.0).
Page 10 it says “hop count is not a concern if the total switching latency is less than the disk I/O timeout value”.
Every switch hop adds latency (frame needs to be put in ASIC, processed then sent on its way).
Switch latency is measured in microseconds.
Disk I/O – is that the same as multipathing timeout? So 60 seconds for MPIO default in Windows?
How are these latencies measured?
Two fairly similar words. One indicates something has a replica or a duplicate to fall back on. The other indicates the strength, can it by itself handle a problem.
Core switches should be equal or higher perf compared to edge switches.
Highest performing switch should be the principal switch.
Redundant links should be placed on different blades/ASICs or at least different port groups.
EHT – edge hold time
New timeout value that can discard blocked frames earlier than the 500ms default (down to 100ms). An I/O retry will still happen for each dropped frame.
Is a new features in FOS 7(confirm) and it is ASIC dependent. Meaning ports in another port group are not affected by the EHT in another port group.
EHT applies to all F_Ports on a switch and all the E_ports that share ASIC with F_Ports.
Intended for initiators only.
Directors interconnected via ICL is not considered a hop in FICON, is it in Open Systems?
Are the links uni-directional?
ICL cables should all have the same length.
ISL can be a bit different, max 30m in difference.
Don’t have ISL and ICL to the same switch/domain.
Hyper-Scale Fabrics: Scale-out Architecture with Brocade DCX 8510 Feature Brief.
Page 15: Brocade recommends core-edge as primary SAN design, or mesh for small fabrics (under 2000). !!! That’s pretty big..
On page 16 it says use full-mesh under 1500 ports.
Fan-In and Fan-Out and Oversubscription
Host ports to Target Ports
Device to ISL
Fan In : number of device ports that need to share a single port, be it target or ISL.
Consider: port queue depth, iops and throughput.
Example: If you have 4 devices with one 8G FC port each (32Gbps) and they are connecting over an ISL of 2x8G to another switch to a storage array that also have 2x8G then there is a 2:1 oversubscription, both on the ISL and on the target ports.
BD consumes switch memory, don’t monitor more than 100 ports on a 48k (no limit on DCX).
Start monitoring a small number of storage ports.
Thresholds and actions are generally different between initiators and targets. Thus place these on different switches.
Monitor Class 3 frame discards (C3TX_TO), they are an indicator of high-latency devices.
Fabric Watch Classes
This is a wide grouping of similar devices.
For example, temperature is a part of the class Environment.
Number of credits: 6+ ((link speed Gb/s * Distance in km) / frame size in KB)
On 8510 4K buffers are available per ASIC to drive 16Gbps to 500km at 2KB frame size. With credit linking, buffers can be borrowed from a neighboring ASIC to extend distance.
Details about ‘credit linking’? Not many hits about this on google.
You can connect DWDMs in pass-thru mode where the switch is providing all the buffering.
FCIP adds a small latency (35 micro seconds). This is without the underlying TCP/IP delays.
Use QoS to give FCIP traffic highest priority.
Use CAR (committed access rate) to limit other traffic.
Use ARL (adaptive rate limiting) and set the limit to the remaining bandwidth.
FCIP traffic believes it is the only one using the bandwidth it has available, other traffic will suffer if they if they are sharing.
Use rate limiting on the FCIP on the Brocade systems, don’t limit it on the IP network.
This is mentioned for when extending mainframe/FICON extension over FCIP.
OC1 =~ 52Mbps or without overhead ~50Mbps
OC12 = 12*52 or about 622Mbps
OC48 = 48*52 or about 2488Mbps
OC12 is recommended for Compression Mode 3 (GZIP/software only)
OC48 is recommended for Compression Mode 2 (SW with HW assist)
Neither of those are recommended for synchronous replications. Mode 0 is recommended and that is HW only compression.
Gaussian or Normal Distribution
There’s quite a bit about new Virtualization Engines in this paper. It basically means a device that has other disk arrays behind it, and then this device presents disks to servers. The danger is told to be that the engine can send a lot of small control frames, using up the buffer credits without using all the available bandwidth.
APM and Fabric Watch can apparently be used to monitor for excessive levels of SCSI reservations. How? – No specific details found but it is apparently threshold configurable in fabric watch.
Less domains equals to reduced:
- inter-switch zone transfers
- name server synchronizations
- RSCN processing
Dynamic Fabric Provisioning (DFP)
Only on Brocade HBAs and 16G.
Dynamically provision switch-generated virtual WWN.
Can be user-generated as well.
WWN stays the same even after HBA replacement.
In practice this means you can zone, QoS even before the HBA is online and before you know what the WWN is of the new device.
BCFD exam is going into Beta testing in January as well!
This post will be updated as I move along through the different objectives / documents.
// Update 2012-01-15: Added the Knowledge Assessment Test.
// Update 2012-01-28: Went through each .pdf and updated some in here.
The link to the Brocade page where it tells you how to register and where to get the material: http://community.brocade.com/docs/DOC-2379
# Note: This link no longer works
When are these available?
On Thursday 12/01/2012 at 0728 EET it was not available.
On Thursay 12/01/2012 at 0803 EET it was available.
So, that would indicate that the time Pearson follows is GMT-6 or Central Time.
On top of that the only available dates for me was 23rd and 24th of January :(
Time to study!
// update, that was changed, it was a mistake so now I get some more time to study :)
Exam Study Resources (page numbers are document page numbers, NOT the PDF page)
As I see it, the importance of each document could be arranged like this:
- CFD 200 Modules 3-7
- SAN Design Best Practices
- FOS Administrator’s Guide
- The rest.
With 1/2 sharing the top spot. I haven’t gone through the modules yet but I presume they all complement each other.
The reason for them sharing the top spot is because for this Beta Exam, the CFD200 material is for 8Gbps (and it has quite a lot of details about the M-series McData switches, which the 16Gbps BCFD did not include).
There is also a Knowledge Assessment on my.brocade.com ‘education’ page.
It’s called “CFD 201 8 Gbit/sec BCFD Knowledge Assessment”. Again, this is for 8G so beware that some stuff may not be up to date if you are doing the Beta for BCFD 16G. But, the actual type of questions is something that is useful. It mentions EFCM or Fabric Manager some times (this is the previous names of DCFM or what’s now called Brocade Network Advisor).
There is a nutshell guide for BCFD, but this is from November 2008 making it possibly even more outdated than the CFD200 material. But, because most of the topics are still valid it would still work as a refresher, but you can’t use it for anything specific.
I am doubtful that the M-series will be included in the BCFD 16G exam but as it’s still in the objectives for the 8G it’s probably wise to not skip that part completely. For that 1.5 years (half 2009 and 2010) when I did SAN support I only had one call about a McData switch.
Exam Study Resources with my comments:
CFD 200 BCFD Design Course Modules 3-7
- Obviously these are the most important. I’ll go through these at a later stage.
Brocade DCX 8510 Backbone Family Datasheet
- Lots of details about the system specs.
SAN Design Best Practices
- Pages 2,5-16,19-26,31,32-36,40-45,51-53,55,58-62,66,67,72
Fabric OS Administrators Guide v7.0
- Pages 37,43,66-70,102,142,151,153,157,196,199,241,273-286,301,314,315,320,372,383,395-398,402-406,414,417,425,429,437,438-443,449,454-461,464,503,504
- 256-area addressing
- WWN-based PID assignment
- enabling/disabling a port and port decommissioning
- gateway links, ICL,
- RADIUS/LDAP authentication
- fddcfg / DCC/SCC policies
- device authentication
- advanced zoning (regular, broadcast, frame redirection, lsan, qos, ti)
- traffic isolation zoning (and VF considerations for TI zones)
- bottleneck detection
- in-flight encryption and compression (technologies, enabling/disabling)
- licensing (enable 10GbE, 7800, QoS, FCIP Extension, FICON acceleration, etc, etc, etc)
- advanced performance monitoring (top talker, frame monitor, end-to-end)
- adaptive networking (ingress rate limiting)
- QoS prioritization (SID/DID or CS_CTL – class specific control)
- trunking (ISL, ICL, EX_Port, F_Port)
- Long Distance (buffer credit allocation, max distance, credit recovery)
- FC-FC Routing (support platforms)
- interopability (FOS vs M-EOS)
Fabric OS Command Reference v7.0
- Pages 302,695,716,721,957,
- fcrconfigure / fcredgeshow
- Why is the test for “Invalid Word Transmission” called ITW?
- Ah, on portThConfig it is called “Invalid Transmission Word”.
Fabric OS FCIP Administrators Guide v7.0
- Pages 1,6
- FCIP platforms and supported features
- 7800, FX8-24 and FR4-18i
- FCIP Trunking
- Adaptive Rate Limiting
- 8G FC Ports
- Compression (LZ and Deflate)
- Acceleration (FCIP Fastwrite, OSTP)
- VLAN Tagging
- Jumo Frames
- 7800 switch hardware overview
- FX8-24 has support for all features above, except: Jumbo frames (only FR4-18i supports those), IPv6 addresses for FCIP tunnels or IPsec, or 3rd WAN optimization hardware (the other do support this pre FOS 7)
- FCIP platforms and supported features
Monitoring and Diagnostic Testing in Today’s High Speed High Density Networks
- Pages 2-4
- powerpoint presentation of four pages in total
- fc cable lengths
- measuring loss
- embedded diagnostics (bottleneck detection, fabric watch, frame monitoring, port fencing)
- fmmonitor is a CLI that you can use to set up frame monitoring, for example SCSI reservations and aborts.
Brocade Network Advisor SAN User Manual
- Pages 12,164,186,255,596,770,794,796
- “Connectivity Map Toolbar” & “Product List”
- Call Home Feature
- Copying and Deleting Views
- SAN Device Configuration (configuration repository management)
- LSAN Zoning
- Performance Overview
- Bottleneck detection
Why dB Loss Matters for Building Reliable Stable Networks
- Pages 2,3
- total 8 pages
- link lengths and link loss budgets
Brocade 6505 Hardware Reference
- Pages 13,15
- ISL trunking
- fos native and AG modes
Brocade Access Gateway Administrator’s Guide
- Pages xiv,72,
- supported hardware and software (which switches and FOS)
- enabling NPIV on M-EOS and Cisco switches
- CISCO: config t; npiv enable
- new features -F_Port static mapping, APM, B6510, Target Aggregation, Direct target attachment, N_Port monitoring
“You can run the agshow command to display Access Gateway information registered with the fabric. When an Access Gateway is exclusively connected to non-Fabric-OS-based switches, it will not show up in the agshow output on other Brocade switches in the fabric.”
- Page xviii
- Supported Hardware: Standalone switch B8000 and the blade FCOE10-24
- IGMP configuring (IGMP is used in multicast, ethernet)
- Replacing the B8000
- and copy running config and stuff! Looks very similar to the Cisco CLI.
Brocade Adaptors Admin Guide
- Pages 3,13,
- AnyIO technology on the 1860 Fabric Adapter, just change the SFP and set the mode with bcu port –mode or bcu adapter –mode.
- HBA or FC mode (FC)
- Ethernet or NIC mode (GbE)
- CNA mode (FCoE)
- Adapter Support (OS + description of adapters)
- AnyIO technology on the 1860 Fabric Adapter, just change the SFP and set the mode with bcu port –mode or bcu adapter –mode.
The New Data Center 1st Edition
- Pages 65,66,78
- Fabric Based Disaster Recovery (64-67)
- An overview of some of the extension technologies and reasons behind them.
- Network Security (77) + Power, Space and Cooling Efficiency (78)
- Network Security is not FC related.
- Fabric Based Disaster Recovery (64-67)
In a moment of frustration/impatience I posted on ‘Brocade Certified’ that it was still 3 weeks until the results are posted. Good boss of training Joe Cannata replied that I could e-mail him and get the results before. I did, and I passed! Most excellent and kind of them to do that! So nice to not have to wait the extra three weeks until it’s entered/updated in the Pearson Vue system. What kind of name is that anyway? Pearson Vue? Latin? Turns out no.
Maybe there’s another meaning behind the name Pearson (besides son of pear).
Back to the subject, how nice was it to find out that I passed? The first few days / week after the exam I was really aching to find out. Then I pretty quickly forgot about it. It wasn’t until maybe a month and a half later that I thought, cmooon why is this so slow! All in all it feels pretty awesome to have the certificate! It took a pretty heavy amount of studying. I think maybe now I got the courage to go for the designer one as well!
OK, did the exam. Took 2.5 hours.
It felt like I was plowing through them pretty fast but after some 80-questions the speed dropped significantly and I started to have trouble focusing. Short unfocus/try to think about something else , stand up a bit helped. This test wasn’t as nerve-wrecking for me as the BCFA – maybe it had something to do with that I didn’t pay for this one.. (smålänningen i mig).
I realized last night that I hadn’t listened to the BCFP nutshell guide (the one with audio) so I quickly went through that and I believe it helped a bit, even though it’s for FOS 6.3 and 8G FC stuff. If you do this exam when it goes live, definitely do listen to the nutshell, maybe several times and don’t forget to click the ‘notes’ button on the slides, there’s more info there. It’s kind of a very sped up/condensed version of the whole material, but there’s some things that aren’t even mentioned that you’ll only find in the material. But it is free. It’s nice to have somebody read it out loud, maybe they phrase it a little different than you do (in the head) that is just what’s needed for you to understand it.
Also there are the simulations/labs in the material when you buy that’s really valuable.
For the BCFA they had virtual class rooms for this nutshell – looks like they might have changed this now and recorded it. One avenue less to ask questions.
Now the waiting begins. Two months until I get to know the results. Yikes!
Gotta find something else to occupy myself with now.. hmm..
Archiving? Buddhism? Apparently BCFD (fabric designer) is beta-tested in Dec/Jan. This would be the next logical step for me. Why not huh? =) Anybody has any ideas? Right now I just feel like getting myself into the next Malaz book by Ian C. Esslemont – Stormwielder, supposedly about Greymane!
How to set up two FCIP tunnels between two B7500: http://community.brocade.com/message/15261
Connecting two sites via FCR: http://community.brocade.com/message/16828#16828
Set up FCIP between two B7800: http://community.brocade.com/message/14216#14216
Why is there an IP in ipaddrshow on FC interfaces: http://community.brocade.com/message/3978#3978
OK, so now you’ve been studying for a while? Read the material once or twice, made notes. Have you read more details about a command in the command reference guide, or more details about a technique in the FCIP Admin Guide or the FOS guide? Or how do you do your studying?
Now would be a good time to head over to the page where you see the requirements/objectives of the exam.
Write them down and think about each. Could you for example ‘Demonstrate knowledge of how to manage FCIP/FC distributed external solutions’ ? No? Why not? What parts about it do you not understand, are there any foggy parts, etc. If you can explain this to somebody, that’s a great achievement (now you just have to find somebody who doesn’t get glazed eyes but listening to you talk about fibre channel).
Maybe if you imagine a presentation (like in school) and you have to tell somebody about this specific feature. What would you go over?
Two weeks left for me now. Just hit me today that it’s 4 hours long and 180+ questions (so about 1min 20seconds per question). My brain is not going to function very well after the test. Gotta stock up on carbs, don’t drink before.
Another studying tip!
Head over to the forum/community at Brocade. Do it. After you read this post :p
For example in the BCFP there will be questions about NPIV, so it makes sense to read the forum for threads regarding NPIV. Right? I mean there’s bound to be troubleshooting, and getting some ‘real’ experience troubleshooting SAN is quite hard to get, especially with stuff like FCIP/FCR.
Example link: http://community.brocade.com/message/18897#18897
You don’t need an account. If you register you don’t get access to much anyway.
Here is one as an example. But there are lots of posts there and there’s a ton to learn. I usually just troll/help out on HP’s Enterprise Server/Storage forum but I think I’ll start reading on this as well.
A bit quiet here.
I’m quite busy at work and also studying for the BCFP.
Currently wrapping up the end of ‘going through the material and putting what I find interesting in a document so that I can print it and re-read it and make notes etc’.
One thing I found was a Top Talker feature called ‘EE Monitors’.
As far as I could tell this was the same as Top Talker in ‘port mode’ in opposed to ‘Top Talkers’ which was ‘Top Talker’ in fabric mode.
Brocade Embedded SAN switches are called B54xx like 5424 etc.
Ingress Port – Traffic Entering a switch port (rx)
Egress Port – Traffic Exiting a switch port (tx)
The free material does not go through the virtual fabrics and the exam objectives does not mention them specifically. Even so – it’s probably a good idea to get some grip about it anyway as it is mentioned in the BCFP 8G material and there are questions concerning it in the Nutshell Guide and the BCFP 8 knowledge assessment. Also it is mentioned in the pre-requisites for the BCFP (the AFS 141).
There is also quite a lot of information about VF in the FOS Administration Guide. You can find this guide in lots of places but it is in the exam objectives in the link above.
It is what it says it is – a way to create independent and logical fabrics and switches that you can use to segment your SAN.
It does not require a license.
There’s logical fabrics and logical switches.
From the FAQ:
A Logical Fabric is an implementation of a Fibre Channel fabric with one or more Logical
Switches participating in the fabric. A Logical Fabric has its own independent instance of
fabric services, name server, zoning database, and so on.
A logical switch needs a fabric id. Default is 128 but can be changed. Same FID cannot be used for same logical switch in the same chassi. You move ports from the default to the new switches. VE_ and EX_ ports needs to be configured after the move. LD, QoS, F_port buffers/trunking may not be enabled on the port.
Max 8 VFs in the DCX, enabling it is disruptive (requires a reboot).
DCX uses 10-bit addressing. Uses part of the last part of the ALPA part of the PID.
Means that that part of the PID does not always indicate a port area.
Increases limit of NPIV, support loop devices.
For 8G products it’s available on the DCX, B5100 and the B5300.
For 16G it’s available on 6510, VA-40FC
FC10-6, FS8-18, FCOE10-24 ports can only be part of the default switch.
(X)ISL — interconnecting switches
The default switch – is the first logical switch you create.
To connect a logical switch (henceforth known as LS) to another one you can just have one of the ports in the LS as an E_port, or you can use XISL – extended ISL.
To use XISL you designate one LS as a base switch. This is used for interconnects and you can have ISLs for several fabrics on this one port/cable. It can have E, VE and EX, VEX ports. *x_ports can only be in the base switch. One base switch per chassi, on DCX platforms the default cannot be the base switch. You connect the base switch to other base switches and then the other logical switches with the same FID merge. By default the logical switches are enabled to use XISL. You can combine normal ISL and XISL. Normal ISL have a lower cost.
ISL (between physical switches)
DISL (between Logical Switches)
IFL (routing, not merging)
XISL (several LISLs inside)
LISL (part of an XISL)
With XISL a logical port is created, their WWN start with 5x.
fosconfig –enable vf
lscfg –create FID [-base] [-force]; setcontext FID; swichdisable (set Domain ID etc); configure; switchenable
lscfg –config 128 -slot <slot> -port <port>
lscfg –delete non-default-logical-switches
lscfg –change 5 –newfid 7 (disables switch and sets it); fosexec –fid FID -cmd “switchenable”
fosexec –fid FID -cmd “cmd” (how to run a command on another LS)
fosexec –fid all -cmd “cmd” (on all logical switches)
ipaddrset -ls 123 –add 10.10.10.10/24 (set an IP for a logical switch, to segment management)
Still studying for Brocade’s BCFP Exam.
This post is to try to put light on some of the terms/technologies you’ll be surrounded by when learning about FCIP.
Guides you should see are the “Fabric OS FCIP Administrator’s Guide” and you should probably start with the material for BCFP – part 4 (theory) and 5 (administration).
Basically the FC frames will be encapsulated in packets over TCP/IP, making the TCP/IP part invisible/irrelevant to the SAN fabric and the FC frames invisible/irrelevant to the TCP/IP. Except of course for the FC routers that bridge the networks. It is possible to run FCR over FCIP as well via the VEX ports (virtual EX_port). Extension. This means that it’s using TCP flow control, no BB credits.
Tunnel (VE_port) – are seen as VE_ports in the fabric.
Circuits (GbE ports) are inside a tunnel (VE_port)
Is a logical connection between two IP addresses.
Metric 0 – active (you can have several links at metric 0)
Metric 1 – standby
FCIP tunnels support max two hops.
On the FX8-24:
1x 10Gbe and 10x 1GbE
Not, all ports at the same time.
Basically adding more circuits to a tunnel, not recommended to set up several tunnels (limited anyway) but because ISL trunking is not supported on VE_ports.
FICON timeout: 1s
FC timeout: 4s
Consider altering these depending on your setup/latencies.
Virtual Fabric considerations
Define several logical switches inside a physical.
You can with FOS 7.0.0 have a VE_port (the GbE ports) defined in the base/default switch and then share it with other logical switches, giving you the possibility to extend/route the fabrics over a shared trunk while they are still isolated. You cannot mix dedicated (in an LS) and a shared (in default) in the same FCIP tunnel.
Not enforced if there is no contention (there is free bandwidth)
VC0 (or F_frames – fabric frames) – always first.
QoS_High: >50% : : 6
QoS_Medium: >30% : 3
QoS_low: >20% : 1
DSCP (6 bits of priorities – 64 )
L2CoS (3 bits of priorities- 8 )
Priority is set in the TOS – in the header.
(four different ones, hardware, software, mix, auto)
“lossless” failover only in FOS 7.0.0. (brocade chipset did not share ports)
You cannot use both 10GbE and get 20GbE. You can have them active/standby or use both A/A and get 5Gbps on each.
Disabling port != failover testing. Can/will cause disruptions.
Crossports are addresses and routes that belong to the other 10GbE (XGE) port’s DP or VE group.
The crossport for xge0 is xge1 and for xge1, the crossport is xge0. To use crossports, the port must be configured in 10 Gbps mode.
The crossport is the non-local XGE port for a VE_Port group. In other words, for VE ports 12 through 21, xge1 is the local XGE port and xge0 is the crossport. For VE ports 22 through 31, xge0 is the local XGE port and xge1 is the crossport.
(selective acknoledgement – prevent that each lost packet requires an ack, bundles up several lost packets into one, default is ON)
Adaptive Rate Limiting
Configure minimum and maximum rates on an FCIP circuit.
Let’s say you have one FCIP router with two circuits going to two independent IP-routers, these two share a link to another site. The idea is that then you can use ARL to configure minimum half of the shared link on each of the circuits from the FCIP router to the IP router, and a max of the whole one. So if one goes down, you’re not stuck with half and you’re not oversubscribing. There, easy to explain in words :d
2 x 10GbE ports, 12 x 1GbE and 12 x FC8
Link to hardware page on Brocade.
6 x 1GbE ports, 16 x FC8
Link to hardware page on Brocade.
- What settings are you going to have on the ports/links/tunnels?
- Configure hw ports (media type, mode etc)
- Disable VE_ports (Virtual FC E_ports) with the tunnel (portdisable)
- Create ip intf for each phy Ethernet port that’s going to be used (portcfg ipif)
- Config IP route for each port to specify an IP Gateway (not required; portcfg iproute)
- Verify IP network between the two IP interfaces that will form the tunnel. (portcmd –ping slot/port)
- Create an FCIP tunnel (circuit 0; portcfg fciptunnel; portcfg fcipcircuit)
- Config FCIP Features (SACK, compression, etc)
- Verify config, enable VE_ports, verify that it’s working
- Add more circuits to the tunnel(s).
FCR; Fibre Channel Routing; FC-FC Routing; etc; etc.
This has many names. FC-FC Routing service provides FCR (fibre channel routing).
Basically what it does is that it lets you zone devices in two separate fabrics without merging them.
These two separate fabrics are called ‘edge fabrics’ in Brocade lingo, they are otherwise known as SAN islands.
The edge fabric is connected to a backbone fabric (an FC router or at least an EX_Port).
Integrated Routing – is a licensed feature that lets you run FCR on a port that is in a normal port in a normal switch or port blade (so not in a dedicated router switch or router blade).
There’s a few things required to set up FCR:
- Verify that you have the proper setup (required licenses/hardware)
- Assign backbone FIDs (switchdisable; fosConfig –disable fcr; fcrconfigure; fosconfig –enable; switchenable)
- Configure FCIP tunnel (not required but: portcfg fciptunnel 8/ge0 create 2 220.127.116.11 18.104.22.168 0 -v 100 -p 3 -P 7 . Remote IP first, tunnel ID, vlan, Classes for layer2 control and data traffic)
- Configure IFLs – inter fabric links – links between edge and backbone fabrics (portcfgvexport, portcfgexport 7 -a 1 -f 30 . port 7, enable, fabric id 30)
- Modify cost on the EX_ports (not required; portdisable; portcfgexport 7 -a 1; fcrrouterportcost 7 10000; for default, set it to 0; fcrRouteShow also shows cost)
- Connect cables (if you do it before and they are configured as E_port you may get segmentation).
- Configure trunking on EX_ports (not required but if you have more than one link, please do, same commands as for E_port trunking)
- Configure LSAN zones (same as normal zoning; zonecreate “lsan_zone_fabric”, “wwn; wwn2; wwn3”; cfgadd “zone_cfg”, “lsan_zone_fabric”; cfgenable “zone_cfg”). Use lsanzoneshow -s. Shows imported/exist/configured/initializing.
fcrphydevshow, fcrproxydevshow are also useful.
- Confirm that it’s working (fcrfabricshow, switchshow, portcfgshow, portexport 7, portshow 7)
A little theory
Front domains -> always there
Translate domains (also xlate domains…). -> only there when devices are online and zoned
The FC router has a pool of wwns and proxy ids that it assigns to devices.
Basically a host that wants to communicate with a target in another fabric communicates with a proxy WWN in its own fabric (so the FC router is like a middle man that passes frames back n forth).
A little more information
There may be mistakes in here but that’s just how it is, I tried to keep it as factual as possible and used several sources.
There will most likely be more of these posts coming up.
More studying. Only a month and a half to go.
Currently repeating/re-reading BCFA stuff and mostly focusing on the new stuff. Expect to do this this whole week but gradually weave in more BCFP stuff.
An idea – I don’t think I need to re-hearse the BCFA that much. The objectives between the two exams are very different and there’s no overlap as far as I can tell. Focusing more on the BCFP now but it was nice anyway to do a short repeat of the BCFA stuff, get back in the game.
I’ve been going through what’s recommended (the material) and these are the useful pages:
Please note that the second item in each list is the actual page number in the document.
I also took the liberty of adding pages before/after in case they were adding context to the page. It’s not like I’m going to try to remember the pages by heart. And quite often the pages referenced by Brocade were just one page in the middle of a chapter.
For example page 63 in FOS Admin Guide 7 is either about setting ipaddr or routing/FC NAT. I think it’s the FC NAT. Page 77 is either for adressing/WWN based PID assignment or lossless DLS. Page 80 is port numbering schemes for various blades or Forward Error Correction. 99 is verifying syslog/audit log or introduction to RADIUS/LDAP. 117-118 is lossless DLS or overview of IP protocols.
Page 3 in the troubleshooting guide is to the document history or one of the pages with list of common symptoms. Also Page 1 in FCIP Admin Guide is probably not the one they meant :)
There’s some really weird ones in the admin guide. For example page 582 does not exist in FOS admin or 132 is empty in FCIP Admin guide , in both real page counters and the numbers on the pages in the book.
Also some starting/ending points/pages are a little strange, why cut it off there and not the whole chapter/section?
From the Brocade Certified group on facebook I saw that these pages numbers were used to writing a/questions on the exam. Guess this explains why some of the pages are odd, maybe they were written down at a previous version of the document or they just don’t want to document everything :) In the same group they claim that the page they used is the one on the actual page, not the one in for example adobe reader. This means the numbers under Real are the ones pointing to the right pages.
Anyway, with the details from http://community.brocade.com/docs/DOC-2041 here we go:
Updated the numbers on FOS Admin guide (2011-08-18)
- Fabric OS Administrators Guide v7.0 (53-1002148-02)
- Pages 63,77,80,99,100,117,118,128,133,137,200,272-281,287-302,372,382,395,404-412,413,418,422-433,435-438,447-481,582
- Real Pages: 102-103, 120, 139, 168, 173-174, 177, 240, (271-303), 311-343, 410, 412, 422, 435, 444-453, 458-478, 487-521,
- Fabric OS Command Reference Guide v7.0 (53-1002147-01)
- Pages 239,244-246,283-290,380-383,609,610,637,653,661-663,701-710,714-717,824,885,930,953-956,1028,1029,1083
- Real Pages: 273, 278-280, 316-324, 643, 644, 671, 687, 695-697, 735-744, 748-751, 857-858, 918-920, 964, 987-990, 1061-1063
- Fabric OS Troubleshooting Guide v7.0 (51-1002150-02)
- Pages 3,22,31,38,92,
- Real Pages: 23, 43-44, 51-52, 112
- Brocade 1860 Datasheet (GA-DS-1566-00)
- Brocade SAN Health Family Data Sheet (GA-DS-870-03)
- Fabric OS v7.0 Release Notes
- Pages 11,12
- Real Pages: 11, 12
- Brocade Network Advisor SAN User Manual 11.1.x (53-1002167-01)
- Pages xxxviii,xxxix,47-52,148,202,230-233,647,648,782,911
- Real Pages: 38-39, 91-96, 192, 246, 274-277, 691-692, 826, 955
- Brocade Network Advisor Installation Guide 11.1.x (53-1002320-01)
- Page 9
- Real Page: 9
- Fabric OS FCIP Administrator’s Guide (53-1002155-01)
- Pages 1,6,29-37,54,111-113,132
- Real Pages: 15, 20-21, 43-52, 68, 125-127
- Access Gateway Administrator’s Guide (53-1002156-01)
- Pages 11,22,52,53,67-69,
- Real Pages: 31, 42, 72-73, 87-89
- Brocade Adapters Administrator’s Guide (53-1001923-01)
- Page 35
- Real Page: 57
- Pre-release CFP 300 Course (unedited material)
- Modules 2-8
I am currently going for the BCFP – fabric professional – exam, but I did the BCFA 6 months ago so I’ll re-read the material and of course there’s the new stuff with FOS 7, new hardware, 16g, new ASIC that I should probably learn as well.
Some new stuff
‘fabric name’ is a new feature. But this is also more usable in VF – which is not part of BCFA. Firmware upgrades are the same (phew).
DCFM is now called Network Advisor and it also has IP/routing and MPLS functionality now.
Of course the 16G blades (with the first 8 ports capable of handling 10GB FC) and the FC10-6 blades.
D_port diagnostics (set a port to this before joining it to a trunk, or use it to measure distance on a long distance link, is accurate up to 5m).
IDLE/ARB fill words are no longer necessary to configure (except on 8G platforms and not on Condor3).
Condor3 is the new ASIC for the 16G blades.
New/larger/longer/better ICL between the new directors that use QSFP instead of the crap max 2m copper cable.
One thing that’s great about the kindle is that you can put the Brocade material on it (even in PDF) – just change viewing mode to landscape/horizontal and it will look great. Two pages per slide. I still have material from my old so that one works. But the material that is given for free now has 0 access rights so it does not work on the Kindle.
The way I write my personal notes is: write them off from the brocade material in my own words.
- I do this on google docs.
- I then download it into .doc and then
- e-mail it as an attachment to email@example.com.
- Then next time you hook up your kindle to wifi it will download the documents, converted to .azw.
What’s important here is to not use lists, as the conversion from a google doc saved as word and then e-mailed to firstname.lastname@example.org does not like lists, it only takes the first level in the list.
I instead used headers, lost of them.
This is also nice because you can put a TOC which is clickable on the kindle.
Also pictures work in this conversation.
Right now I’m reading through the BCFA material and writing what I find useful in a google docs document – later I will take this and send to my Kindle – for some of that memorization :)
Think it’s a good idea to refresh some of the BCFA stuff before tackling the BCFP material, it was about 6 months ago since I took the BCFA and there are of course the new 16G hardware that was available then.
If you’re new to all this – do check out the FC 101 training by Brocade, it is pretty awesome and I think I’ll listen/read it once or probably more before the exam. It’s good to refresh the fundamentals.
After the announcement of the available material Fabric OS 7.0.0a has been released.
On the page http://community.brocade.com/docs/DOC-2041 only the first revision of the 7.0 release notes is available.
The BCFA 16G beta course material is also available, probably a good idea to read up on both and do a little rehearsing.
The old attachments are these:
- 1860_FabricAdapter_DS.pdf (942.3 K)
- FOS_TrblShoot_v700-02.pdf (1.8 MB)
- FOS_CmdRef_v700.pdf (4.6 MB)
- FOS_AdminGd_v700-02.pdf (7.2 MB)
- FOS_FCIP_AdminGd_v700.pdf (2.8 MB)
- NetworkAdvisor_SAN_InstallGd_v1110.pdf (450.3 K)
- NetworkAdvisor_SAN_Manual_v1110.pdf (9.5 MB)
- v7.0.0_releasenotes_v1.0.pdf (1.5 MB)
- SH_Family_DS_03.pdf (157.7 K)
- Brocade_Adapters_v22.214.171.124_Admin_Guide.pdf (1.5 MB)
- AccessGateway_AdminGd_v700.pdf (919.0 K)
With my notes:
- 1860_FabricAdapter_DS.pdf (this is just a brochure/data sheet of that product)
- FOS_TrblShoot_v700-02.pdf (trbl and diag guide, 3 June 2011, 138p)
- FOS_CmdRef_v700.pdf (cmd reference manual- 29 April 2011, 1132p)
- FOS_AdminGd_v700-02.pdf (3 June 2011, 580p)
- FOS_FCIP_AdminGd_v700.pdf (29 April 2011, 136p)
- NetworkAdvisor_SAN_InstallGd_v1110.pdf (13 May 2011, 47p)
- NetworkAdvisor_SAN_Manual_v1110.pdf (13 May 2011, 1301p)
- v7.0.0_releasenotes_v1.0.pdf (29 April 2011, 125p)
- SH_Family_DS_03.pdf (SAN Health Family, brochure/data sheet)
- Brocade_Adapters_v126.96.36.199_Admin_Guide.pdf (27 October 2010, 292p)
- AccessGateway_AdminGd_v700.pdf (29 April 2011, 102p)
The newer versions are not available if you have a registered account on my.brocade.com without any product registered to it. I’ve e-mailed Brocade asking for more access. In the meantime some are available on for example HP.com – but the Brocade release notes are not available there. And the HP Release notes for 7.0.0a have the HP names of the products.
On HP’s site you can go to the manuals for the 8/40 to get some more documents.
- v7.0.0a_release notes_HP in pdf (HP’s customer notice about the release of 7.0.0a)
- FOS_TrblShoot (3 June 2011, the same)
- 7.x.x FOS_CMD_reference guide (not available)
- 7.x.x FOS Admin Guide (3 June 2011, the same)
- 7.x.x AG Admin Guide (not available)
- 7.x.x. Fabric Watch Admin Guide (29 April 2011)
- Neither is FCIP admin guide available
- But the Network Advisor stuff is. (assume these are the same)
- 7.0.0 Message Reference (this has the public error messages that you may see in errdump)
Actually it looks like the documents available are quite up to date. I guess the Admin Guides etc don’t update that often anyway. There is however a document in the manuals link above that explains that there have been some changes to some of the Brocade Documents. This might be worth checking out.
The conclusion of this little exercise is that the material available is currently good enough (for me).
The problematic PDF
The CFP300 material on http://community.brocade.com/docs/DOC-2041 is encrypted so that it cannot be printed/re-edited without a password.
If you try to open this with evince (default .pdf viewer in Gnome) it will ask for a password.
pdftotext (comes with the software suite poppler) says:
Error: Weird encryption info
Error: Incorrect password
It’s only the material starting with M0* that has this issue, this has also been seen with other documents. Maybe this is because they were created with a too new version of Adobe Acrobat that evince/pdftotext doesn’t support.
The rest of the material are going to be public and they are user/admin guides anyway. But the M0* files are from the actual course material for the 16G so this is why.
The solution on RHEL6 x64: install FoxitReader. Download the .rpm – then hit ‘rpm -Uvh FoxitReader-1.1-0.fc9.i386.rpm’ and it will be installed. To start it just hit ‘FoxitReader’.
Anyway I think it’s nice of Brocade to pre-release the course material for those doing the beta-test. If you want the real material the cheapest is 650$ and then you get the material, narration of the pdfs (usually good quality, not just reading off the presentations) and a few quite good lab exercises.
Just threading along here with the material, slowly but steady.
I’m starting with the NPIV / Access Gateway stuff. It’s a bit more complicated than just a switch that isn’t its own domain, it’s also mapping the virtual WWN to the N_ports (a switch in AG mode has N_ports that connect to F_ports in another switch). Usually N_ports are on hosts’ and targets’ ports and the switches’ has the F_ports.
Just signed up for this test!
Time to start studying!
Details for what is needed: http://community.brocade.com/docs/DOC-2041
Also, it’s free – get it while it’s hot ;)
How to find the latest firmware for an HP Brocade SAN switch:
How to find out which Brocade Firmware is the latest and which is the last supported one?
- On HP SPOCK: http://h20272.www2.hp.com/ (this required an HP Passport, they are free to create).
- Go to Switches, then click on B-series connectivity stream.
The B-series conenctivity stream document gets updated often and it has the recommended (usually latest) and the supported Fabric OS releases for each switch type.
How to find the last supported firmware for an HP Brocade SAN switch:
Usually the EOL of a firmware is announced when a new one is released.
See this customer notice of 6.4.1b:
After a question in my SAN switch firmware upgrade article I made a comparison of two downloads of 6.3.1b (one via IBM and one from HP) – the only differences were a file called ancillary and one called EULA.pdf. I used examdiff to find the differences.
All the sub-directories were the same, only the above two files were added in the HP one.
I believe quite strongly that you can use the HP firmwares to upgrade Brocade switches that are branded by other vendors.
At least IBM and normal Brocade ones.
As they are using the very same Brocade firmware that Brocade themself use, it might be hard for the vendors to change the switch that much.
It would be interesting to investigate if other vendors add something to make theirs not, but I have no way of acquiring such a firmware.
The EULA looks like a normal HP standard end user license agreement form. The HP ancillary.txt file contains this:
“This ancillary.txt file provides information as to how to obtain the open source or other third party licenses in this distribution. To obtain such licenses, run the following CLI command at the prompt, “opensource”.
This ancillary.txt file also provides the instructions for customers who require a copy of the
machine-readable GPL Source Code by written request. Upon your written request, HP will provide to You, for a fee covering the cost of distribution, a complete machine-readable copy of the GPL Source Code. Your written request for GPL Source Code can be sent via email to FC_Infrastructure_OpenSourceRequest@hp.com. In the request, include product name, version number, your name, and your shipping address. “
This is my guide/template to upgrading Fabric OS (FOS) – Firmware – on the Brocade SAN Switches. If you have any additions, comments or questions please go ahead and comment or if you have any questions you can find my e-mail on https://guldmyr.com. The post has been updated over 188 times according to my WordPress revisions, first update in January 2011.
This article was originally built from my experience with HP branded Brocade SAN Switches – not with any other OEM or pure Brocade switches. I have however since beginning this document gotten experience with other vendors.
I do not think others are different except for licenses and some default fabric.ops.
I made a comparison of two downloads of the 6.3.1b Fabric OS Firmware (one via IBM and one from HP). You can find a link to the “IBM” firmware and release notes after 6.x in that article too. I found that they are very similar and the HP firmware works on the IBM switch and vice versa. Another example is that firmware gotten from HDS works on an HP branded Brocade switch.
When you see 7.2.x this means any version in the Fabric OS 7.2.x series. For upgrades, this would generally mean the latest available in that series (like 7.2.1g for 7.2.x or 8.0.2d for 8.0.x) unless of course there is a problem with the latest. Sign up for your vendor’s security and update alerts to get notified about new releases.
Carefully plan the upgrade, it takes time but it is rewarding and worth it.
Updates in this article:
2017-03-10: 8.x has been out for a while
2017-04-29: new links to Brocade FOS target path and better links for where to fetch firmwares
2017-05-03: 8.0.2b and added links to Upgrade Guides for 8.0.0 and 7.4.0)
2018-01-01: 8.1.2a and 7.4.2b
2018-02-08: 7.4.2c and 8.0.2d
2018-06-15: 8.2.0a and 8.1.2d and 8.0.2e
2018-10-11: fixing some links, Brocade is now Broadcom so some links are not working anymore surprise. Some HP links no longer work so removed those too.
2018-12-07: 7.4.2d and 8.0.2f and 8.1.2f
2019-04-18: FOS Target Path on Broadcom. From a reader got a link to NetApp's Brocadeassist portal where newer firmware can be found than on IBM's. Some more link updates and link to 8.2.1b
2019-04-19: More link fixes. FOS 6.x - 8.x firmwares can all be downloaded from the brocadeassist portal.
2019-04-20: 8.1.2g and 8.2.1b on IBM release notes so updated links
2019-07-17: BlueChris in a comment found a nice HPE link to older Firmwares!
2019-08-23: 8.2.1c is on NetApp's link but there are no release notes - I'd pass2
2019-11-21: all links to firmware downloads don't work right now. Only one I have found is to HPE, but there you need support contract. Found a public link? Help out other people in the comments please. I have a cache myself up to 7.4.x but I'm not sure about the legalities.
2020-05-10: people are sharing links in the comments. Many thanks! Also some new releases, 8.2.2 is out.
2020-09-27: 9.0.0a got released in August
One major release at a time is required for the upgrades after 5.2.x, see details below at the release notes section.
If you have to upgrade many steps, you should upgrade to the latest in the series (or if it's very new, probably safest to go with the second newest, just check the release notes of the newest to make sure nothing related is fixed).
If the switch is on 5.1.x you can go directly to 5.3.x.
What I usually recommend is this path:
5.0.1d -> 5.2.3 -> 5.3.2c -> 6.0.1a -> 6.1.2c -> 6.2.2g -> 6.3.2e -> 6.4.3h > 7.0.2e > 7.1.2b > 7.2.1g > 7.3.2a > 7.4.2f > 8.0.2f > 8.1.2g > 8.2.2b > 9.0.0a
It's also possible to upgrade from a version earlier than 6.4.1b to 7.0.x or from 7.0.x to 7.2.x - but this is a disruptive upgrade (meaning ports will go offline/online during upgrade).
Brocade now has a document that describes a process of determining the 'ideal' version of Fabric OS you should be running. It is called Brocade FOS Target Path.
Yet one more official document to help is the Brocade Fabric OS Features and Standards Support Matrix, 8.2.x
There is also a section (Recommended Migration Paths to FOS ) in the release notes describing how to get to the release you're reading notes for. In addition to these, there are Upgrade Guides from Brocade, at least for newer Fabric OS ( 7.4.0 and 8.0.0).
There are newer releases being released every now and then, in several series at the same time. You can think of it as releasing updates for Windows XP and 7 at the same time.
For example, in February 2011 6.4.1a and 6.2.2e were released by HP. You can see this on HP's site if you look at the date next to the download. Quite often Fabric OS versions are not released by the OEMs at the same time, for example "Customer Notice of 7.1.0a release 25th of March 2013" HP released 7.1.0a before IBM.
Which is the recommended one? Usually it's the latest one in the highest series that the switch supports. If you have storage from more than one vendor you may want to check with all and see if they all support the version you want to upgrade to. Vendors certify their equipment with different firmware versions. If you have a tape library, ask the vendor if they have a recommended / list of certified versions.
HP: HP B-series Connectivity stream (available in HP SPOCK).
Brocade: "Brocade FOS Target Path"
Other: Contact them for their compatibility matrices, for example IBM, HDS, EMC, Fujitsu.
Brocade also has their own "Brocade Fabric OS 7.x Compatibility Matrix" which lists compatibility with other vendors.
You could in principle also say that (some blades in directors are excepted from these generalizations):
2G cannot upgrade to Fabric OS 6.x
4G and 8G can be on Fabric OS 6.x
All 4G except some 4/8 & 4/16 (that's 200E) and HP's P- and C-class 4G blade switches (4012 & 4024) can run 6.4.x
8G can run Fabric OS 6.4.x
8G and above can run Fabric OS 7.x
16G (Gen5) needs to be on Fabric OS 7.x or Fabric OS 8.x
32G (Gen6) and 64G (Gen7) needs to be on Fabric OS 8.x or Fabric OS 9.x
Do you want to use the latest one in each series? Probably.
Do check for published advisories and the release notes in the firmwares.
Some models or blades may work on 7.0.x and not on 7.1.x or vice versa.
Fabric OS 7.3.x supports all hardware that supports 7.2.x.
Basically you need to read the release notes for at least the version you are upgrading to, to confirm that it supports your switch.
Download firmware links:
- 5.x and 6.x at HPE's http://whp-aus2.cold.extweb.hp.com/pub/softlib/software12/COL22074/co-86832-6/FOS-Drawer_Statement.htm
- New Link for some older 5.3-8.0.x from HPE: https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX-fd389e31ce584b35911249126f
- 6.0.x can also be found here: ftp://ftp.hp.com/pub/softlib/software10/COL22074/
- For FOS 7.x
- For FOS 8.x
- NetApp's Broadcom/Brocade link: http://www.brocadeassist.com/public/NetAppRelease
- This has more documents and newer firmware than the IBM one.
If you go to downloads for HP's 4/16 there is a link that also takes you to the older FOS firmware. If you don't click through it also only have the firmware that this switch supports. So the latest on there at the moment is 6.2.2f.
On the link above you can also download HP's branded NA (Network Advisor, previously known as DCFM - Data Center Fabric Manager), see notes about that below.
If you click on manuals on the left side you will also be able to download release notes and other guides and references.
5.0.x firmware can also be found at http://ftp.hp.com/pub/softlib/software12/COL22074/co-86832-6/FOS-Drawer_Statement.htm
6.x, 7.x and 8.x. can be found in the IBM and NetApp links.
Firmware Upgrade Order
You also probably want to decide on an order to upgrade the firmware on the switches.
It's possible to do it via DCFM (now called Network Advisor, used to be something else) one switch at a time or even in parallel. I'd advice against doing it in parallel. One at a time and one step at a time seems the most cautious one. It's not too bad to run a SAN with switches in different firmwares. One idea is to have all switches of one model on the same firmware. If you need to upgrade in several steps, do one step at a time.
Also, switches that are of higher importance like Principal Switch, Core Switches or Seed Switches for DCFM/NA. Should you start with these or perhaps start with another switch of less importance to make sure the upgrade goes smoothly?
With more recent firmwares (6.4 and 7.x) it's possible to jump more than one hop - if you are ok with disruptions in the network. Nice if you need to upgrade switches that aren't in production.
Brocade release notes in .pdf
Notes from the release notes:
Upgrading from Fabric OS 5.0.x to 5.2.3 is supported
Upgrading from Fabric OS 5.1.x to 5.3.1a is supported, but upgrading from Fabric OS 5.0.x or a previous release directly to 5.3.1a is not.
Upgrading to Fabric OS 6.0.0b is only allowed from Fabric OS 5.3.x. (6.0.0c is a special upgrade version, only meant to be used in between firmware upgrades)
Upgrading to Fabric OS 6.1.2c is allowed only from Fabric OS 6.0.0b
Upgrading to Fabric OS 6.2.2f is allowed only from Fabric OS 6.1.0a or later.
Upgrading to Fabric OS 6.3.2e is allowed only from Fabric OS 6.2.0a or later.
Upgrading to Fabric OS 6.4.3f is allowed only from Fabric OS 6.3.x. You can upgrade non-disruptively from 6.2
Upgrading to Fabric OS 7.0.2 can be done non-disruptively from Fabric OS 6.4.1a or later.
Upgrading to Fabric OS 7.1.2 can be non-disruptively upgraded from 7.0.x and 7.1.x. With caveats: For example, any previously existing error log entries with FOS v7.1.0 will be permanently lost once upgraded to FOS v7.1.2.
Upgrading to Fabric OS 7.2.x can be done non-disruptively from 7.1.x. Disruptively from 7.0.x is supported.
Upgrading to Fabric OS 7.3.x can be done non-disruptively from 7.2.x. Disruptively from 7.1.x is supported (see the FOS_UpgradeGuide_v730.pdf and the Brocade Release notes).
Upgrading to Fabric OS 7.4.x can be done non-disruptively from 7.3.x. From 6.4.x with firmwarecleaninstall
Upgrading to Fabric OS 8.0.x can be done non-disruptively from any Brocade 16G (Gen 5) platform and all blades in the Supported blades table running any FOS v7.4 firmware. From 7.3.0 with "firmwaredownload -s"
Upgrading to Fabric OS 8.1.x can be done non-disruptively from Brocade platform running 8.0.2 or later. From 7.4.x disruptively with "firmwaredownload -s".
Upgrading to Fabric OS 8.2.x can be done non-disruptively from Brocade platform running 8.1.0a or later. From 8.0.x disruptively with "firmwaredownload -s".
Any Brocade platform listed in the Supported Device section running any FOS 8.2 version can be non-disruptively upgraded to FOS 9.0.0
About non-disruptively: This means you can go to 7.0.xfrom earlier than 6.4.1a but ports will go offline during the upgrade.
See the release notes or Upgrade Guides for more details.
DCFM: Data Centre Fabric Manager / BNA: Brocade Network Advisor .
From 6.2.2a release notes:
With the introduction of Fabric OS 6.1.1, certain features and functions were removed from Web Tools (resident in the firmware) and migrated to the DCFM management application. HP recommends that, before you upgrade to Fabric OS 6.1.1x or later, if DCFM is not running on your fabric, you review the Web Tools functionality moved to DCFM, page 29 in these release notes and take note of what has changed so you can assess the impact on your fabric.
Fabric OS 7.x cannot be managed by DCFM 10.4 or BNA 11.0. You need BNA 11.1.0, see the release notes for 7.x.
Brocade Network Advisor 12.4.0 or later is required to manage switches running FOS 7.4.0 or later.
Brocade Network Advisor 14.0.1 or later is required to manage switches running Fabric OS 8.0.1 or later
Updates to documents
Sometimes Brocade releases updates to the manuals without actually updating the manuals. On HP's page you can find them as "Documentation Updates", "Fabric OS Administrator's Guide Update".
Fabric Watch and MAPS with FOS v7.3
Users running Fabric Watch for switch monitoring in FOS v7.3 are advised to convert to MAPS monitoring before upgrading to FOS v7.4. If you don't, Fabric Watch will stop working.
Also the APM have been replaced with Fabric/Flow Vision.
See the release notes of the firmware for the specifics. For example Fabric OS 8.0.2 cannot be in the same fabric as for example HP C-Class 4/12 FC switches (4024) and one must use Fibre Channel Routing.
Download old Brocade Fabric OS Firmware.
Basically, you need to update in steps.
To get FOS 5.2.1b and 6.0.0c firmware: Contact OEM Vendor or Brocade. I've found that two vendors have the firmware available online for free: HP and IBM, see below:
Eventually after looking around on HP's old pages we found to http://ftp.hp.com/pub/softlib/software12/COL22074/co-86832-6/FOS-Drawer_Statement.htm - this link sometimes changes.
Link to IBM's page for downloading FOS 6 firmwares. This has firmwares going back all the way to FOS 2.6, it even has Fabric OS 6.0.0c and 5.2.3. On the page they have listed release notes and a little further down there is a link called "Release 6 Firmware".
Actually, if you click on 'Release 6 Firmware' you are taken to a page on brocade.com where you can find many different firmwares, including 5.x and 7.x
IBM also have a link about FOS 7.x and FOS 8.x
Also note that some features does not exist/work on older Fabric OS. For example on Fabric OS 5.1.x DHCP and SCP may not work (which forces you to use static IP and ftp).
Equivalent Product Names
Page with the equivalent Brocade and HP product names.
Page with the model number as seen in switchshow and HP's model and Brocade's model. This is a good one.
Page for correlating IBM and Brocade product names.
HP recommend that you upgrade one fabric and one switch at a time.
Waiting a week or at least a couple of days after you upgrade the first fabric is a good idea - gives you time to see if anything went wrong, if you can fix it and if you can do anything different next time.
See HP SPOCK for more details in regards to compatibility and interop modes.
The HP B-series Connectivity Stream lists the recommended firmware and all the supported ones for each switch model. It also has a list of the supported SFPs. Find it by clicking on "Switches" in the left-hand navigation pane under the "Other Hardware" section. The Connectivity Stream is great and it is updated often so I will not link directly to it. You need an HP Passport to log on to HP SPOCK - it is free to create and you do not need a contract or product in warranty.
Other vendors have similar matrices. HP for example does not have a list stating which Fabric OS firmware is supported with which HP P6000 firmware. The idea is that you go with the general recommendation of Fabric OS firmware.
Do read the release notes for the firmware(s) you decide on: for example not all 4GB SAN-switches can run the 6.4.x FOS. The 8- and 16-port 4Gbps switches (Brocade 200E) do not run 6.4.x or 6.3.x.
Only 8Gb and 16Gb switches can run the 7.x.x FOS.
The release notes also have the fixes, enhancements, upgrade paths and supported switches.
Generally the Brocade versions of the release notes are more verbose when it comes to fixes, but if you have an HP branded it might be easier to use the HP one as that has the HP names of the products. Also it might be hard to find the Brocade release notes if you do not have a contract with Brocade. Other vendors (like IBM/Fujitsu/HDS) provide you with the Brocade version of the release notes. You can find the release notes from their support pages.
Do consider updating OS patches, HBA drivers/firmware, management softwares and storage drivers/firmware. For example Qlogic had driver updates to their drivers that prevent HP blades from getting stuck in G_port after a reboot. Another for Qlogic FC cards was to not write a partition table on Dell servers at 2TB on the LUN (not so nice for > 2TB disks)..
SANLoader is an unofficial HP tool to upgrade firmwares. With this you do not have to create an ftpserver etc. Contact HP Support, they may give this to you.
This is meant to be used when the other ways does not work, but it helps out a lot as you do not have to set up an FTP/SCP server.
Sanloader used to (winter 2010) not work well on Windows 7 and may not work flawlessly on the pre 6.x firmwares.
- Set up a ftp/scp server and upgrade via the CLI (command line interface).
- Use DCFM ( Data Center Fabric Manager - now called Network Advisory ) to upgrade firmware.
- Firmware can also be upgraded through the web interface (click on switch admin and then on firmware download). You will still need an FTP/SCP server for this though. See the web tools admin guide page 73-74 (FOS 6.2.x but it hasn't moved).
FileZilla is a free FTP-server that works well. There are many alternatives around. But unfortunately some don't work sometimes (not 100% sure but probably combination of older FOS with older ftp client with FTP server that couldn't handle that client) as listed in the comments thread in this post. FileZilla is however still on sourceforge so you may want to be careful about installing that - it might contain malware. Storing them on a Synology NAS works - thanks Henny!
For FTP clients:
- /usr/bin/ftp in Ubuntu (also in Ubuntu on Windows)
- WinSCP for a free opensource Windows alternative that does both ftp and SCP (and more).
For SCP any machine with Linux and sshd on should work. You can also get an scp-server running on Windows, OpenSSH would work. Both protocols are old, SCP is safer while FTP is sending data in clear text.
Personally I like doing this via the CLI. The Network Advisor way gives you the possibility to upgrade in parallel, but that's also risky. If you use a Linux server to provide the firmwares via SCP don't forget to let the switches in via firewall or tcp.wrapper ( /etc/hosts.allow ). If you do the upgrade via ftp - make sure that passive and active ftp both works.
How to access the SAN-switch
The most common way is to access the CLI of a Brocade switch by connect to the IP of it with an ssh- or telnet-client, PuTTy is the name of a free Windows client. If you are comfortable with CLI, Windows 10 has WSL and a good ssh and scp client built in. Telnet is unsafe so do try to use the ssh at all costs. Windows 10 has Bash which is in my opinion much nicer to use than putty.
It's also possible to access the switch CLI via a serial cable, however as the firmware files are several 100MB (approaching 1GB for 6.4.x) that's not really viable when upgrading firmware. Hyperterminal is a free windows tool that comes by default in some Windows versions. You can also use PuTTy for serial access.
To access the web interface just point the web browser to . It requires Java. The Java version that's supported is listed in the release notes of the Fabric OS.
Here on HP's Support Forum are some more notes about v6.x. Basic steps:
Note: version 6 does not require to specify the exact folder location SWBDxx: it just needs the root containing "the install" file
1) Unpack the downloaded firmware in the FTP or SCP download directory
2) Start the FTP/SCP Server and allow access
3) Connect to the CLI of the switch via telnet or ssh
4) Type this in the CLI: firmwaredownload
5) Answer all questions: when it asks for File Name be sure to write /v6.4.1b, that is the folder under which you find all the SWBDxx folders. Failing to do so makes it impossible to download the firmware
6) Wait for reboot of the switch and reconnect, check the firmware version with the "version" command
More notes about the upgrade
CLI Command to start the update process is firmwaredownload - this starts the interactive version, it is possible to specify user, directory, host directly via the CLI. See the Command Reference Guide for details. There are reference guides for each major Fabric OS release.
Please use forward slashes when specifying directories.
For example when you unzip the firmware file and it creates a sub-folder in the FTP-root that is called v5.3.1a then you need to specify /v5.3.1a as the directory.
For firmwares prior to 5.3.x you have to specify the release.plist - /v5.2.2a/release.plist.
However it says in the release notes for 5.2.3 that release.plist is no longer needed.
In some cases you may have to specify the sub directory.
For example the 4/16 HP Switch is a Brocade 200E with switchtype 34. So you would then use directory SWBD34 - /v5.3.1a/SWDB34. You can also try with /v5.3.1a/release.plist, /v5.3.1a/SWDB34/release.plist or /v5.3.1a/install. However with 5.3.1a you should not have to so /v5.3.1a should be enough.
switch:admin> firmwaredownload Server Name or IP Address: IP.TO.SCP.SERVER User Name: username File Name: /path/to/v6.2.2e Network Protocol(1-auto-select, 2-FTP, 3-SCP) : 3 Password: Server IP: IP.TO.SCP.SERVER, Protocol IPv4 Checking system settings for firmwaredownload... System settings check passed. You can run firmwaredownloadstatus to get the status of this command. This command will cause a warm/non-disruptive boot on the switch, but will require that existing telnet, secure telnet or SSH sessions be restarted. Do you want to continue [Y]: y Firmware is being downloaded to the switch. This step may take up to 30 minutes. Preparing for firmwaredownload... Start to install packages... dir ################################################## [[lots of these for all packets]] ################################################## [[also stuff like these are seen many times:]] warning: /etc/fabos/pki/switch.0.rootcrt created as /etc/fabos/pki/switch.0.rootcrt.rpmnew kernel-module-ipsec ################################################## Removing unneeded files, please wait ... Finished removing unneeded files. All packages have been downloaded successfully. Firmware has been downloaded to the secondary partition of the switch. HA Rebooting ...
Transfer Protocol and Connectivity
If you are using SCP and that does not work, please try with FTP. If neither works, see if something else can log on to the FTP/SCP server. And of course, make sure the right permissions/root directory are set on the FTP-server. If your FTP/SCP server has log files, check them. If it works from one client but not from the switch, check the logs and see if there's a difference. Sometimes if the SCP doesn't work via CLI it might work by doing SCP (but starting it from the Web Tools, thanks Eric in the comments for this!).
If you are logged on as root on the SAN-switch you can use the scp- or ssh-client on the switch to confirm connectivity, like this:
ssh username@server ls /tmp/v6.0.1a to list the /tmp/v6.0.1a on the SCP server.
You need to be root to run the above command.
If that also does not work, you have some kind of networking problem - you can try direct connecting a laptop to the LAN interface of the switch. To see the network settings on the switch: ifmodeshow and ipaddrshow
Sometimes when upgrading from 6.1.1d to 6.2.2 we have seen that the passwords have gotten reset.
Default password is then "password" or "fibranne".
You can reset the password with the CLI command "passwd admin" to reset password on the admin account.
If you forget all passwords it might be possible to be able to reset it via the serial cable interface while booting the switch.
On EMC branded switches the default password might be: Serv4EMC
If your switch is out of warranty/contract and it's still working. I'd suggest making a copy(dd in linux for example) of the CF-card. Then if the CF card decides to fail you can just get a new one from random_electronic store and dd the contents of the flash back.
When replacing a switch make sure that the licenses are correct.
If for example you have a switch with 'power pack' - then for HP there is a special spare part number for a switch with power pack and one without. Power pack is a grouping of licenses, which licenses are in the pack differs between models.