Tag Archives: SAN

Setup a 3 Node Lustre Filesystem

Introduction

Lustre is a filesystem often used by clusters because many computers can mount the filesystem simultaneously.

This is a small log/instruction for how to setup Lustre in 3 virtualized machines (one metadata server, one object storage server and one client).

Basic components:

VMWare Workstation
3 x CentOS 6.3 VMs.
Latest Lustre from Whamcloud

To use Lustre your kernel needs to support it. There’s a special one for server and one for the client. Some packages are needed on both.

Besides lustre you’ll need an updated version of e2fsprogs as well (because the version that comes from RHEL6.3 does not support large partitions).

Starting with the MDS. When the basic OS setup is done will make a copy of that to use for OSS and Client.

Setup basic services.

Install an MDS

This will run the MDT – the metadata target.

2GB RAM, 10GB disk, bridged networking, 500GB for /boot, rest for / (watch out, it may create a really large swap). Minimal install. Setup OS networking (static ip for servers, start on boot, open port 988 in firewall, possibly some for outgoing if you decide to restrain that too), run yum update and setup ntp. Download latest lustre and e2fsprogs to /root/lustre-client, lustre-server and e2fsprogs appropriately (x86_64). Lustre also does not support selinux, so disable that (works fine with it in enforcing until time to create mds/mdt, also fine with permissive until it’s time to mount).
Put all hostnames into /etc/hosts.
Poweroff and make two full clones.
Set hostname.

Install an OSS

This will contain the OST (object storage target). This is where the data will be stored.

Networking may not work (maybe device name changed to eth1 or eth2).
You may want to change this afterwards to get the interface back to be called (eth0). A blog post about doing that.

Install a client

This will access and use the filesystem.

Clone of the OSS before installing any lustre services or kernels.

Install Lustre

Before you do this it may be wise to take a snapshot of each server. In case you screw the VM up you can then go back :)

Starting with the MDS.

Installing e2fsprogs, kernel and lustre-modules.

Skipping debuginfo and devel packages, installing all the rest.

yum localinstall \ 
kernel-2.6.32-220.4.2.el6_lustre.x86_64.rpm kernel-firmware-2.6.32-220.4.2.el6_lustre.x86_64.rpm \
kernel-headers-2.6.32-220.4.2.el6_lustre.x86_64.rpm \
lustre-2.2.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64.rpm \ 
lustre-ldiskfs-3.3.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64.rpm \
lustre-modules-2.2.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64.rpm

The above was not the order they were installed. Yum changed the order so that for example kernel-headers was last.

yum localinstall e2fsprogs-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-debuginfo-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-devel-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-libs-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-static-1.42.3.wc3-7.el6.x86_64.rpm \
libcom_err-1.42.3.wc3-7.el6.x86_64.rpm \
libcom_err-devel-1.42.3.wc3-7.el6.x86_64.rpm \
libss-1.42.3.wc3-7.el6.x86_64.rpm \
libss-devel-1.42.3.wc3-7.el6.x86_64.rpm

After boot, confirm that you have lustre kernel installed by typing:

uname -av

and

mkfs.lustre --help

to see if you have that and

rpm -qa 'e2fs*'

to see if that was installed properly too.

By the way, you probably want to run this to exclude automatic yum kernel updates:

echo "exclude=kernel*" >> /etc/yum.conf

After install and reboot into new kernel it’s time to modprobe lustre, start creating MDT, OST and then mount things!
But hold on to your horses, first we ned to install the client :)

 

And then the Client

Install the e2fsprogs*

We cannot just install the lustre-client packages, because we run a different kernel than the ones that whamcloud have compiled the lustre-client against.

We can either back-pedal and install an older kernel. Or we can build (from source / SRPMS) a lustre-client that works on a kernel of our choosing. The later option seems like a better way, because we can then upgrade the kernel if we want to.

 

Build custom linux-client rpms

Because of a bug it appears that some ext4 source packages are needed – while they are not. You need to add some parameters to ./configure. This will be the topic of a future post.

The above rpmbuild should create rpms for the running kernel. If you want to create rpms for a non-running kernel you are supposed to be able to run.

Configure Lustre

Whamcloud have good instructions. Don’t be afraid to check out their wiki or use google.

/var/log/messages is the place to look for more detailed errors.

On the MDS

Because we do not have infiniband you want to change the parameters slightly for lnet to include tcp(eth0). These changes are not reflected until reboot (quite possibly something else) – but just editing a file under /etc/modprobe.d/ called for example lustre.conf is not enough.

Added a 5GB disk to the mds.

fdisk -cu /dev/sdb; n, p, 1, (first-last)

modprobe lustre lnet

mkfs.lustre –mdt –mgs

mount

On the OSS

Also add the parameters into modprobe.

mkfs.lustre –ost

mount

On the client

Add things into modprobe.

mount!

Write something.

Then hit: lfs df -h

To see usage!

 

Get it all working on boot

You want to start the MDS, then the OSS and last the client.
But while it’s running you can restart any node and eventually it will start working again.

Fstab on the client:
ip@tcp:/fsname /mnt lustre defaults,_netdev 0 0

Fstab on the OSS and MDS:
/dev/sdb1 /mnt/MDS lustre defaults,_netdev 0 0

While it’s running you can restart any node and eventually it will start working again.

Brocade Accredited Server Specialist – BASP

http://www.brocade.com/education/certification-accreditation/accredited_server_connectivity/curriculum.page

I’m currently preparing for yet another accreditation, the previous one I took was the BADCS,

The BASP (Brocade Accredited Server Specialist) appear to focus on the server side. Things like:

  • how to install drivers
  • HBA management tools
  • describe features
  • how to run diagnostics

This accrediation has the most questions of all the current ones, but it has the same amount of time allotted (one hour) so this exam will have a lot less time available for each question.

The curriculum for this accreditation are also free, they are called Introduction to HBA and Introduction to CNA. There’s also some docs about the 1860 Fibre Adapter. They can be found on Brocade’s Saba/training website under my.brocade.com.

 

// Update 20140422: This accrediation has been replaced with something else. See the current list here: http://www.brocade.com/education/certification-accreditation/index.page?

Brocade Accredited Data Center Specialist – BADCS

 

Time to study for another one :) Working my way towards the “Data Center Track”. To complete it it would be enough for me to complete 5 accreditations.

This one has a pretty cool name – BADCS!

http://www.brocade.com/education/certification-accreditation/accredited-data-center-specialist/prerequisites.page

I haven’t tried one of these Accredited exams before, but as far as I can tell:

  • Cheap: only 20$ USD
  • The exam is web based, no need to find a test center, you can do it exactly when you want to.
  • Accreditations do not expire
  • You don’t _have_ to take the course in the prerequisites before taking the exam, but it is recommended :)

Also, for this Accreditation the pre-requisite is the FC-101 course on brocade’s SABA page – and it’s free!

– The BADCS exam consists of 38 questions and lasts 60 minutes
– To pass this exam you must get a score of 71% or better 

So that’s about 27 correct out of 38 questions.

The objectives are on this page.

The only part I was initially not entirely sure about is the “Given a scenario, describe when portlog dumps are required”. The objectives indicate that a Fibre Channel theory knowledge is necessary, so the FC-101 course seems like a very good idea to study. I doubt many people remember specific FC mechanisms/theory if they don’t work with these occasionally. Like the well-known addresses – who remembers the address of the name-server or controller? =)

My general tip for the BADCS: Learn the material of the FC-101 course. Really. Learn. it.

You may be tricked into thinking that Brocade’s accrediations are easy because you can do them from home.

Access Gateway – NPV – TR

Say what??

Access Gateway – Brocade

NPV (N_port Virtualization (not NPIV) – Cisco

Transparent Mode – QLogic

These are all names for the basic idea / functionality but as there’s no standard the vendors have made up their own names for it.

A switch in Access Gateway (AG) mode does not consume Domain IDs, you can do port mapping, needs NPIV on the port in the switch that it connects to. AG requires a switch / fabric to connect to as it doesn’t run the normal fibre channel services.

It is very useful in case you are going to mix vendors in your fabric. Meaning you can populate the core with Brocade switches and then connect other vendors’ switches in the above modes to the Brocade switches.

On some QLogic switches you can also set a port into TR-mode, see this post on HP’s EBC forum about how to do it. It is not exactly the same as AG or NPV, because you still need to do zoning on the QLogic switch.

There is also the IPM by Qlogic for IBM – it looks like a module that you cannot switch between ‘fabric’ and ‘IPM’ mode. Which is what you can do on a Cisco or on a Brocade switch.

 

Storage FC HBA Transfer Size Tuning

HP just published an advisory describing how to tune some parameters for Emulex, Qlogic and Brocade Fibre Channel HBAs: c02518189. It sounds like these are new, but these changes have been around for at least 6 months in all three vendors’ HBAs.

Emulex

“Emulex driver version 2.42.002 or later, along with OneCommand Manager version 5.1.53.2 or later,”

Use HBAnywhere to change these.

Examples to tune the server or port level transfer size:

  • 128 kbytes, set the LimTransferSize = 2 and ExtTransferSize = 0 (default)
  • 512 kbytes, set the LimTransferSize = 0 (default) and ExtTransferSize = 0 (default)
  • 1 Mbytes, set the LimTransferSize = 0 (default) and ExtTransferSize = 1

Qlogic

This is part of the Qlogic SANSurfer utility.

  • c:\>qlfc -tsize /fc
  • c:\>qlfc -tsize /fc /set 128
  • c:\>qlfc -tsize /fc /set default

Brocade

  • bcu drvconf –key bfa_max_xfer_len –val 64
  • bcu drvconf –key bfa_max_xfer_len –val 128

BCFP – good threads in the brocade forum

How to set up two FCIP tunnels between two B7500: http://community.brocade.com/message/15261

Connecting two sites via FCR: http://community.brocade.com/message/16828#16828

Set up FCIP between two B7800: http://community.brocade.com/message/14216#14216

Why is there an IP in ipaddrshow on FC interfaces: http://community.brocade.com/message/3978#3978

 

 

 

 

BCFP – more studying

OK, so now you’ve been studying for a while? Read the material once or twice, made notes. Have you read more details about a command in the command reference guide, or more details about a technique in the FCIP Admin Guide or the FOS guide? Or how do you do your studying?

Now would be a good time to head over to the page where you see the requirements/objectives of the exam.

Write them down and think about each. Could you for example ‘Demonstrate knowledge of how to manage FCIP/FC distributed external solutions’ ? No? Why not? What parts about it do you not understand, are there any foggy parts, etc. If you can explain this to somebody, that’s a great achievement (now you just have to find somebody who doesn’t get glazed eyes but listening to you talk about fibre channel).
Maybe if you imagine a presentation (like in school) and you have to tell somebody about this specific feature. What would you go over?

Two weeks left for me now. Just hit me today that it’s 4 hours long and 180+ questions (so about 1min 20seconds per question). My brain is not going to function very well after the test. Gotta stock up on carbs, don’t drink before.

BCFP – 16G Studying

Another studying tip!

Head over to the forum/community at Brocade. Do it. After you read this post :p

For example in the BCFP there will be questions about NPIV, so it makes sense to read the forum for threads regarding NPIV. Right? I mean there’s bound to be troubleshooting, and getting some ‘real’ experience troubleshooting SAN is quite hard to get, especially with stuff like FCIP/FCR.

Example link: http://community.brocade.com/message/18897#18897

You don’t need an account. If you register you don’t get access to much anyway.

Here is one as an example. But there are lots of posts there and there’s a ton to learn. I usually just troll/help out on HP’s Enterprise Server/Storage forum but I think I’ll start reading on this as well.

BCFP – VF – Virtual Fabrics

The free material does not go through the virtual fabrics and the exam objectives does not mention them specifically. Even so – it’s probably a good idea to get some grip about it anyway as it is mentioned in the BCFP 8G material and there are questions concerning it in the Nutshell Guide and the BCFP 8 knowledge assessment. Also it is mentioned in the pre-requisites for the BCFP (the AFS 141).

Brocade’s overview page of Virtual Fabrics.

 FAQ on Virtual Fabrics

 

There is also quite a lot of information about VF in the FOS Administration Guide. You can find this guide in lots of places but it is in the exam objectives in the link above.

The theory

It is what it says it is – a way to create independent and logical fabrics and switches that you can use to segment your SAN.
It does not require a license.

There’s logical fabrics and logical switches.

From the FAQ:
A Logical Fabric is an implementation of a Fibre Channel fabric with one or more Logical
Switches participating in the fabric. A Logical Fabric has its own independent instance of
fabric services, name server, zoning database, and so on.

A logical switch needs a fabric id. Default is 128 but can be changed. Same FID cannot be used for same logical switch in the same chassi. You move ports from the default to the new switches. VE_ and EX_ ports needs to be configured after the move. LD, QoS, F_port buffers/trunking may not be enabled on the port.

Max 8 VFs in the DCX, enabling it is disruptive (requires a reboot).

DCX uses 10-bit addressing. Uses part of the last part of the ALPA part of the PID.
Means that that part of the PID does not always indicate a port area.
Increases limit of NPIV, support loop devices.

Hardware

For 8G products it’s available on the DCX, B5100 and the B5300.
For 16G it’s available on 6510, VA-40FC
FC10-6, FS8-18, FCOE10-24 ports can only be part of the default switch.

 

(X)ISL — interconnecting switches

The default switch – is the first logical switch you create.

To connect a logical switch (henceforth known as LS) to another one you can just have one of the ports in the LS as an E_port, or you can use XISL – extended ISL.

To use XISL you designate one LS as a base switch. This is used for interconnects and you can have ISLs for several fabrics on this one port/cable. It can have E, VE and EX, VEX ports. *x_ports can only be in the base switch. One base switch per chassi, on DCX platforms the default cannot be the base switch. You connect the base switch to other base switches and then the other logical switches with the same FID merge. By default the logical switches are enabled to use XISL. You can combine normal ISL and XISL. Normal ISL have a lower cost.

ISL (between physical switches)
DISL (between Logical Switches)
IFL (routing, not merging)
XISL (several LISLs inside)
LISL (part of an XISL)

With XISL a logical port is created, their WWN start with 5x.

CLI

fosconfig –enable vf
lscfg –create FID [-base] [-force]; setcontext FID; swichdisable (set Domain ID etc); configure; switchenable
lscfg –config 128 -slot <slot> -port <port>
lscfg –delete non-default-logical-switches
lscfg –show
lscfg –change 5 –newfid 7 (disables switch and sets it); fosexec –fid FID -cmd “switchenable”

fosexec –fid FID  -cmd “cmd” (how to run a command on another LS)
fosexec –fid all -cmd “cmd” (on all logical switches)
ipaddrset -ls 123 –add 10.10.10.10/24 (set an IP for a logical switch, to segment management)

 

SAN Primer – Introduction to Data Storage

You may have heard about this storage or SAN stuff, but what is it? Is it complicated and cool? Yes. Now it doesn’t have to be complicated, but it sure can be sometimes.
This post is just a brief primer/introduction to storage and what it entails. In case maybe you got a job interview or just would like to know a little bit more about it.

I’ll update this post as I go, last update 2012-07-13 – added some books and free pdfs and links.

What is a SAN?

‘Storage Area Network’ – or storage network.
Generally it doesn’t have to be a ‘network’ it could just be direct connected equipment or peer 2 peer. But what it always entails is a shared storage, most often disk or tape.

What is in a SAN?
When it comes to disk storage on fibre channel there’s a few standard components: FC HBA in the server, SFP and cables, SAN-switch, SFP and cables, FC port in the disk array controller and then there’s something behind the controller that connects disks.

You can connect the FC HBA directly to the disk array.

What is storage?
It’s somewhere where you can store data. Most common today would be: hard drives, flash drives (ssd), magnetic media (tape) and optical media (dvd/blueray/cd).  In a computer you cannot fit hundred of hard drives, but sometimes there is an application that requires lots and lots of data (maybe for example CAD drawings, video editing). This is when a SAN comes in, with only the help of for example a fibre channel card you can give a server access to lots of storage.

How do you do it?
If you want to give a server disk space from a fibre channel SAN this is what you do:

  1. Fullfil the hardware requirements (so fibre channel HBA+drivers and multipath software, SAN-switch, disk array and sfps + cables)
  2. On the SAN-switch create a zone with the disk array’s and the FC HBA’s domain id, port id or port wwn. It’s possible to do it without zones, but they are good for fault isolation.
  3. On the disk array you should now see the server/host, create a disk and map/present it to the host.
  4. On the host you most likely need to do a rescan/reinitialize of the fc-bus.
  5. After the server sees the LUN it will have a new hard disk available, you can use your normal partitioning/format/filesystem tools to create some usable space.

Can I use the same disk on two servers?
This is a pretty common question, the answer is sometimes and the sometimes depends on which file system you are using. It needs to support that more than one host can access it at the same time. NTFS does not support this and if you try it anyway you’ll corrupt the file system. For Windows you need to look into CSV – clustered shared volumes or other networked file systems like NFS/CIFS.

What is the difference between fibre channel and iscsi?

FC is sending SCSI commands over fibre channel, it’s not always fibre or optical cables.
While iSCSI is sending SCSI commands over TCP/IP.
FC is a whole network technology while iSCSI is running on top of a network technology -> TCP/IP.

Some literature:

Both the IBM and the HP one are quite lengthy. The HP one has a lot of HP specific guides, best practices and supported configurations. The FC 101 by Brocade actually goes quite deep into the theory of the FC protocol.

BCFP – Brocade Certified Fabric Professional 16G Beta Exam

More studying. Only a month and a half to go.

Currently repeating/re-reading BCFA stuff and mostly focusing on the new stuff. Expect to do this this whole week but gradually weave in more BCFP stuff.

An idea – I don’t think I need to re-hearse the BCFA that much. The objectives between the two exams are very different and there’s no overlap as far as I can tell. Focusing more on the BCFP now but it was nice anyway to do a short repeat of the BCFA stuff, get back in the game.

The Material

I’ve been going through what’s recommended (the material) and these are the useful pages:
Please note that the second item in each list is the actual page number in the document.
I also took the liberty of adding pages before/after in case they were adding context to the page. It’s not like I’m going to try to remember the pages by heart. And quite often the pages referenced by Brocade were just one page in the middle of a chapter.

For example page 63 in FOS Admin Guide 7 is either about setting ipaddr or routing/FC NAT. I think it’s the FC NAT. Page 77 is either for adressing/WWN based PID assignment or lossless DLS. Page 80 is port numbering schemes for various blades or Forward Error Correction. 99 is verifying syslog/audit log or introduction to RADIUS/LDAP. 117-118 is lossless DLS or overview of IP protocols.
Page 3 in the troubleshooting guide is to the document history or one of the pages with list of common symptoms. Also Page 1 in FCIP Admin Guide is probably not the one they meant :)

There’s some really weird ones in the admin guide. For example page 582 does not exist in FOS admin or 132 is empty in FCIP Admin guide , in both real page counters and the numbers on the pages in the book.

Also some starting/ending points/pages are a little strange, why cut it off there and not the whole chapter/section?

From the Brocade Certified group on facebook I saw that these pages numbers were used to writing a/questions on the exam. Guess this explains why some of the pages are odd, maybe they were written down at a previous version of the document or they just don’t want to document everything :) In the same group they claim that the page they used is the one on the actual page, not the one in for example adobe reader. This means the numbers under Real are the ones pointing to the right pages.

Anyway, with the details from http://community.brocade.com/docs/DOC-2041 here we go:

Updated the numbers on FOS Admin guide (2011-08-18)

  • Fabric OS Administrators Guide v7.0 (53-1002148-02)
    • Pages 63,77,80,99,100,117,118,128,133,137,200,272-281,287-302,372,382,395,404-412,413,418,422-433,435-438,447-481,582
    • Real Pages: 102-103, 120, 139, 168, 173-174, 177, 240, (271-303), 311-343, 410, 412, 422, 435, 444-453, 458-478, 487-521,
  • Fabric OS Command Reference Guide v7.0 (53-1002147-01)
    • Pages 239,244-246,283-290,380-383,609,610,637,653,661-663,701-710,714-717,824,885,930,953-956,1028,1029,1083
    • Real Pages: 273, 278-280, 316-324, 643, 644, 671, 687, 695-697, 735-744, 748-751, 857-858, 918-920, 964, 987-990, 1061-1063
  • Fabric OS Troubleshooting Guide v7.0 (51-1002150-02)
    • Pages 3,22,31,38,92,
    • Real Pages: 23, 43-44, 51-52, 112
  • Brocade 1860 Datasheet (GA-DS-1566-00)
  • Brocade SAN Health Family Data Sheet (GA-DS-870-03)
  • Fabric OS v7.0 Release Notes
    • Pages 11,12
    • Real Pages: 11, 12
  • Brocade Network Advisor SAN User Manual 11.1.x (53-1002167-01)
    • Pages xxxviii,xxxix,47-52,148,202,230-233,647,648,782,911
    • Real Pages: 38-39, 91-96, 192, 246, 274-277, 691-692, 826, 955
  • Brocade Network Advisor Installation Guide 11.1.x (53-1002320-01)
    • Page 9
    • Real Page: 9
  • Fabric OS FCIP Administrator’s Guide (53-1002155-01)
    • Pages 1,6,29-37,54,111-113,132
    • Real Pages: 15, 20-21, 43-52, 68, 125-127
  • Access Gateway Administrator’s Guide (53-1002156-01)
    • Pages 11,22,52,53,67-69,
    • Real Pages: 31, 42, 72-73, 87-89
  • Brocade Adapters Administrator’s Guide (53-1001923-01)
    • Page 35
    • Real Page: 57
  • Pre-release CFP 300 Course (unedited material)
    • Modules 2-8

BCFA – Brocade Certified Fabric Administrator 16G Beta

I am currently going for the BCFP – fabric professional – exam, but I did the BCFA 6 months ago so I’ll re-read the material and of course there’s the new stuff with FOS 7, new hardware, 16g, new ASIC that I should probably learn as well.

Some new stuff

‘fabric name’ is a new feature. But this is also more usable in VF – which is not part of BCFA. Firmware upgrades are the same (phew).
DCFM is now called Network Advisor and it also has IP/routing and MPLS functionality now.
Of course the 16G blades (with the first 8 ports capable of handling 10GB FC) and the FC10-6 blades.
D_port diagnostics (set a port to this before joining it to a trunk, or use it to measure distance on a long distance link, is accurate up to 5m).
IDLE/ARB fill words are no longer necessary to configure (except on 8G platforms and not on Condor3).
Condor3 is the new ASIC for the 16G blades.
New/larger/longer/better ICL between the new directors that use QSFP instead of the crap max 2m copper cable.

Kindle

One thing that’s great about the kindle is that you can put the Brocade material on it (even in PDF) – just change viewing mode to landscape/horizontal and it will look great. Two pages per slide. I still have material from my old so that one works. But the material that is given for free now has 0 access rights so it does not work on the Kindle.

The way I write my personal notes is: write them off from the brocade material in my own words.

  1. I do this on google docs.
  2. I then download it into .doc and then
  3. e-mail it as an attachment to youraccount@free.kindle.com.
  4. Then next time you hook up your kindle to wifi it will download the documents, converted to .azw.

What’s important here is to not use lists, as the conversion from a google doc saved as word and then e-mailed to youraccount@free.kindle.com does not like lists, it only takes the first level in the list.

I instead used headers, lost of them.

This is also nice because you can put a TOC which is clickable on the kindle.

Also pictures work in this conversation.

BCFP 16g Exam Studying

Right now I’m reading through the BCFA material and writing what I find useful in a google docs document – later I will take this and send to my Kindle – for some of that memorization :)

Think it’s a good idea to refresh some of the BCFA stuff before tackling the BCFP material, it was about 6 months ago since I took the BCFA and there are of course the new 16G hardware that was available then.

If you’re new to all this – do check out the FC 101 training by Brocade,  it is pretty awesome and I think I’ll listen/read it once or probably more before the exam. It’s good to refresh the fundamentals.

BCFP 16G Beta Exam Material

After the announcement of the available material Fabric OS 7.0.0a has been released.

On the page http://community.brocade.com/docs/DOC-2041 only the first revision of the 7.0 release notes is available.

The BCFA 16G beta course material is also available, probably a good idea to read up on both and do a little rehearsing.

The old attachments are these:

  • 1860_FabricAdapter_DS.pdf (942.3 K)
  • FOS_TrblShoot_v700-02.pdf (1.8 MB)
  • FOS_CmdRef_v700.pdf (4.6 MB)
  • FOS_AdminGd_v700-02.pdf (7.2 MB)
  • FOS_FCIP_AdminGd_v700.pdf (2.8 MB)
  • NetworkAdvisor_SAN_InstallGd_v1110.pdf (450.3 K)
  • NetworkAdvisor_SAN_Manual_v1110.pdf (9.5 MB)
  • v7.0.0_releasenotes_v1.0.pdf (1.5 MB)
  • SH_Family_DS_03.pdf (157.7 K)
  • Brocade_Adapters_v2.3.0.0_Admin_Guide.pdf (1.5 MB)
  • AccessGateway_AdminGd_v700.pdf (919.0 K)

With my notes:

  • 1860_FabricAdapter_DS.pdf (this is just a brochure/data sheet of that product)
  • FOS_TrblShoot_v700-02.pdf (trbl and diag guide, 3 June 2011, 138p)
  • FOS_CmdRef_v700.pdf (cmd reference manual- 29 April 2011, 1132p)
  • FOS_AdminGd_v700-02.pdf (3 June 2011, 580p)
  • FOS_FCIP_AdminGd_v700.pdf (29 April 2011, 136p)
  • NetworkAdvisor_SAN_InstallGd_v1110.pdf (13 May 2011, 47p)
  • NetworkAdvisor_SAN_Manual_v1110.pdf (13 May 2011, 1301p)
  • v7.0.0_releasenotes_v1.0.pdf (29 April 2011, 125p)
  • SH_Family_DS_03.pdf (SAN Health Family, brochure/data sheet)
  • Brocade_Adapters_v2.3.0.0_Admin_Guide.pdf (27 October 2010, 292p)
  • AccessGateway_AdminGd_v700.pdf (29 April 2011, 102p)

The newer versions are not available if you have a registered account on my.brocade.com without any product registered to it. I’ve e-mailed Brocade asking for more access. In the meantime some are available on for example HP.com – but the Brocade release notes are not available there. And the HP Release notes for 7.0.0a have the HP names of the products.
On HP’s site you can go to the manuals for the 8/40 to get some more documents.

Actually it looks like the documents available are quite up to date. I guess the Admin Guides etc don’t update that often anyway. There is however a document in the manuals link above that explains that there have been some changes to some of the Brocade Documents. This might be worth checking out.

The conclusion of this little exercise is that the material available is currently good enough (for me).

Reading encrypted/password protected pdf on Linux

Brocade Logo

The problematic PDF

The CFP300 material on http://community.brocade.com/docs/DOC-2041 is encrypted so that it cannot be printed/re-edited without a password.

If you try to open this with evince (default .pdf viewer in Gnome) it will ask for a password.
pdftotext (comes with the software suite poppler) says:

Error: Weird encryption info
Error: Incorrect password

It’s only the material starting with M0* that has this issue, this has also been seen with other documents. Maybe this is because they were created with a too new version of Adobe Acrobat that evince/pdftotext doesn’t support.
The rest of the material are going to be public and they are user/admin guides anyway. But the M0* files are from the actual course material for the 16G so this is why.

The solution on RHEL6 x64: install FoxitReader. Download the .rpm – then hit ‘rpm -Uvh FoxitReader-1.1-0.fc9.i386.rpm’ and it will be installed. To start it just hit ‘FoxitReader’.

Anyway I think it’s nice of Brocade to pre-release the course material for those doing the beta-test. If you want the real material the cheapest is 650$ and then you get the material, narration of the pdfs (usually good quality, not just reading off the presentations) and a few quite good lab exercises.

The Studying

Just threading along here with the material, slowly but steady.
I’m starting with the NPIV / Access Gateway stuff. It’s a bit more complicated than just a switch that isn’t its own domain, it’s also mapping the virtual WWN to the N_ports (a switch in AG mode has N_ports that connect to F_ports in another switch). Usually N_ports are on hosts’ and targets’ ports and the switches’ has the F_ports.

Latest FOS ( Fabric OS ) Firmware on HP’s Brocade SAN switches

How to find the latest firmware for an HP Brocade SAN switch:

How to find out which Brocade Firmware is the latest and which is the last supported one?

  1. On HP SPOCK: http://h20272.www2.hp.com/ (this required an HP Passport, they are free to create).
  2. Go to Switches, then click on B-series connectivity stream.

The B-series conenctivity stream document gets updated often and it has the recommended (usually latest) and the supported Fabric OS releases for each switch type.

How to find the last supported firmware for an HP Brocade SAN switch:

Usually the EOL of a firmware is announced when a new one is released.
See this customer notice of 6.4.1b:
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02766238

HP’s Brocade firmwares compatible with other switches?

After a question in my SAN switch firmware upgrade article I made a comparison of two downloads of 6.3.1b (one via IBM and one from HP) – the only differences were a file called ancillary and one called EULA.pdf. I used examdiff to find the differences.

All the sub-directories were the same, only the above two files were added in the HP one.
I believe quite strongly that you can use the HP firmwares to upgrade Brocade switches that are branded by other vendors.

At least IBM and normal Brocade ones.

As they are using the very same Brocade firmware that Brocade themself use, it might be hard for the vendors to change the switch that much.

It would be interesting to investigate if other vendors add something to make theirs not, but I have no way of acquiring such a firmware.

The EULA looks like a normal HP standard end user license agreement form. The HP ancillary.txt file contains this:

“This ancillary.txt file provides information as to how to obtain the open source or other third party licenses in this distribution. To obtain such licenses, run the following CLI command at the prompt, “opensource”.
This ancillary.txt file also provides the instructions for customers who require a copy of the
machine-readable GPL Source Code by written request.  Upon your written request, HP will provide to You, for a fee covering the cost of distribution, a complete machine-readable copy of the GPL Source Code. Your written request for GPL Source Code can be sent via email to FC_Infrastructure_OpenSourceRequest@hp.com. In the request, include product name, version number, your name, and your shipping address. “

Brocade SAN Switch Firmware Upgrades

Overview

This is my guide/template to upgrading Fabric OS (FOS) – Firmware – on the Brocade SAN Switches. If you have any additions, comments or questions please go ahead and comment or if you have any questions you can find my e-mail on http://guldmyr.com. The post have been updated over 173 times according to my wordpress revisions, first update in January 2011.

This article was originally built from my experience with HP branded Brocade SAN Switches – not with any other OEM or pure Brocade switches. I have however since beginning this document gotten experience with other vendors.
I do not think others are different except for licenses and some default fabric.ops.
I made a comparison of two downloads of the 6.3.1b Fabric OS Firmware (one via IBM and one from HP). You can find a link to the “IBM” firmwares and release notes after 6.x in that article too. I found that they are very similar and the HP firmware works on the IBM switch and vice versa. Another example is that firmware gotten from HDS works on an HP branded Brocade switch.

When you see 7.2.x this mean any version in the Fabric OS 7.2.x series. For upgrades this would generally mean the latest available in that series (like 7.2.1g for 7.2.x or 8.0.2d for 8.0.x) unless of course there is a problem with the latest. Sign up for your vendor’s security and update alerts to get notified about new releases.

Carefully plan the upgrade, it takes time but it is rewarding and worth it.

Updates in this article:

Old Updates

2011-02-22: Updated links because the release notes I had before to 6.1.x and above did not work anymore. Also changed the sub-versions in 6.1.x and above to the latest released one by HP.
2011-02-24: Found link to 5.2.x and 6.0.x FOS on HP.com with the help of an ITRC thread.
2011-04-21: Added links for correlating Brocade Product name, model number and HP name. Latest in 6.4.x series is now 6.4.1b
2011-05-05: Added link to Web Tools for 6.2.x with reference to how to upgrade Firmware via the web tools.
2011-05-15: A note added about compatibility regarding the ‘HP’ firmware files and other vendors – as far as I can tell the ones downloaded from HP will work on other non-HP switches. Also posted a new blog post about that. Added link to IBM.com – for correlating product names and for getting (all Fabric OS) firmwares. EMC also has Brocade products.
2011-05-18: Added a link to a post on HP’s support forum where the post helped a bit. Also made post a little easier, wrote a little about the release.plist confusion.
2011-05-24: Added example to show that driver updates are important. Some more restructuring of the article.
2011-07-12: Added FOS 7.0.0a
2011-07-14: Added link to HP knowledge base and updated a link to an ITRC forum thread to point to the new forum.
2011-09-29: Added FOS 7.0.0b and section about CF cards.
2011-10-19: Wrote a bit about firmware upgrade order.
2011-10-24: The HP links to 6.0.0c and 5.3.x seems to not work anymore. I could not find either of these for download on HP’s website. The IBM one still has 6.0.0c (release notes anyway).
2011-12-05: Went through all links to make sure they worked. Re-wrote some of the steps and re-ordered so that ‘decide’ is before ‘prepare’. Added output from switch when doing the firmware upgrade via CLI.
2011-12-10: Added table of contents via a plugin.
2012-01-02: Added FOS 7.0.0c
2012-01-09: Added EMC branded switches default pw
2012-02-14: Added HP’s link to FOS 5.x. firmware.
2012-02-15: Added IBM’s link to FOS 7 info and downloads.
2012-02-21: Some notes about which switches can do which firmware. Re-wrote a part of the upgrade order section.
2012-02-27: Note about licenses.
2012-02-29: Added note about 5.1.x to 5.3.x, made upgrade path clearer. Also made how to find 5.3.x and 6.x firmwares a little clearer for HP’s page.
2012-03-01: Added 6.2.2f and 7.0.1 and note about plist/ftp for 5.1.x
2012-04-03: Addeed 6.3.2e
2012-04-24: Added 7.0.1a
2012-04-27: Rewrote some part of the upgrade section.
2012-06-07: Added 6.4.3 and 7.0.1b
2012-06-14: Added link to Brocade FOS Target Path in decide section.
2012-10-27: Some grammar updates and 16G FOS 7.x requirement. 6.4.3b and 7.0.2.
2012-11-05: Updated links to release notes. Perhaps it’s time to condense the updates list. Notes about passive/active ftp, ifmodeshow|ipaddrshow and java version required (listed in release notes).

2013-03-10: 7.0.2b and 6.4.3c added some notes about compatibility. Improved list of which FOS works with which FC speeds.
2013-03-29: Added 7.1.0a and 7.0.2c. Only HP is out with 7.1.0a as of now. Brocade may have it non-publicly, at least I cannot see it in my brocade. Other minor updates.
2013-04-04: Added link to 6.4.3d
2013-05-02: Updated link to FOS Target Path.
2013-06-23: Changed some ftp:// links to http://
2013-07-16: Added link to IBM’s pdf with pictures for firmware upgrade.
2013-08-03: 6.4.3e by IBM – not available by HP yet. Disruptive upgrades are OK from 6.2 to 6.4.
2013-08-05: Added 7.1.1, updated some links to release notes.
2013-10-03: Made it a bit clearer regarding which is the earliest firmware you can upgrade from. Newer revisions of some Brocade release notes. 7.0.2d out and 6.4.3e link to hp.com
2013-11-14: Removed comment that B300 does not support 6.4.x – it does! It should have been the 200E! Thanks Eugene :)

2014-02-07: Added new link to HP’s page for FOS 5.2 to 6.3  Thanks Leo R!

2014-02-11: Added 7.2.0b and 7.1.1c (HP have 7.1.1c release notes up but IBM does not – to find Brocade version go to IBM’s download the firmware page that’s on Brocade.com and get the release notes from there). Also 3 years anniversary on this post on 2014-02-01!
2014-02-17: But be careful with 7.2.0b – IBM has a note on their 7.x page about 7.2.0b saying: “IBM recommends that customers not deploy FOS 7.2.0b if virtual switch capability is needed. Virtual switch users should migrate to an earlier version as soon as possible.
2014-03-17: The problem with 7.2.0b was likely DEFECT000491192 fixed in 7.2.0c and later also DEFECT000494570 was fixed in 7.2.0d. 7.2.0x seems a bit unstable at the moment. 7.2.1 is currently available for download via HP’s pages but not via IBM/Brocade’s. Also no release notes available. Archived 2013 updates.
2014-04-18: Added 6.4.3f and 7.1.1c and 7.1.2 is out. Updated migration paths a bit. The Brocade release notes of 7.1.x actually have a decent list of the migration path needed now. See the section “Recommended Migration Paths to FOS v7.1.2”.
2014-08-06: 7.2.1a, 7.1.2a, 7.0.2e. Updated some links that had gone bad (FOS Target Path) and made the “Show/Hide” Updates work again.
2014-09-28: 7.1.2b and 7.2.1b is out.
2014-10-01:  7.3.0a is out but can’t find any release notes for it.
2014-10-25: 7.2.1c
2015-01-18: 7.2.1d, 7.3.0c. “HP’s” release notes are too hard to find.. Added a note about FileZilla being a good ftp server. Thanks Harry Redl!
2015-02-20: 6.2.2g, 6.4.3g
2015-04-06: 7.3.1a from HP
2015-04-21: 7.3.1a from IBM
2015-06-19: Note about FileZilla being hosted off sourceforge – installer might contain malware.
2015-07-07: 7.4.0a, 7.3.1b, 7.2.1e. Removed link to HP’s webkey license page. Doesn’t work anymore. Note about BNA version required to manage Fabric OS v7.4 switches.
2015-08-31: 6.4.3g and 7.3.1b new revision of release notes. 7.2.1f, 7.4.0b
2015-09-01: 7.3.1c
2015-09-16: New versions of release notes.
2015-10-25: 7.4.1 is out for the brave and 6.4.3h (Fixes to OpenSSL CVE-2015-0286, CVE-2015-0288, CVE-2015-0289, CVE-2015-0292)
2015-12-05: 7.2.1g and 7.3.1d. Updated some links. Need to go through a lot of the HP ones here to point to HPE..
2016-02-24: 7.4.1b
2016-03-15: 7.3.2
2016-05-17: 7.4.1c
2016-08-13: 7.3.2a, 7.4.1d
2016-12-19: 7.3.2b, 7.4.1e

New Updates

2014-02-01: As of 2014-02-01 HP does not allow anybody without a valid support agreement to download firmwares. Release notes and at least some firmware links appears to still be working. Expect difficulty and broken links while hunting for firmwares. Fabric OS firmwares downloaded from IBM’s site works on HP switches too, but there might be some differences (although I couldn’t find any important ones when I compared 6.3.1b). So far it seems this restriction of access to firmwares only applies to HP servers.

2017-03-10: 8.x has been out for a while
2017-04-29: new links to Brocade FOS target path and better links for where to fetch firmwares
2017-05-03: 8.0.2b and added links to Upgrade Guides for 8.0.0 and 7.4.0)
2017-05-13: 7.4.2
2017-08-07: 8.1.0c
2017-11-09: 8.1.1a
2017-12-05: 8.0.2c
2018-01-01: 8.1.2a and 7.4.2b
2018-02-08: 7.4.2c and 8.0.2d
2018-06-15: 8.2.0a and 8.1.2d and 8.0.2e
2018-10-11: fixing some links, Brocade is now Broadcom so some links are not working anymore surprise. Some HP links no longer work so removed those too.

Steps

  1. decide
  2. prepare
  3. upgrade

Decide

One major release at a time is required for the upgrades after 5.2.x, see details below at the release notes section.

If you have to upgrade many steps, you should upgrade to the latest in the series (or if it’s very new, probably safest to go with the second newest, just check the release notes of the newest to make sure nothing related is fixed).

If the switch is on 5.1.x you can go directly to 5.3.x.

What I usually recommend is this path:
5.0.1d -> 5.2.3 -> 5.3.2c -> 6.0.1a -> 6.1.2c -> 6.2.2g -> 6.3.2e -> 6.4.3h > 7.0.2e > 7.1.2b > 7.2.1g > 7.3.2a > 7.4.2c > 8.0.2d > 8.1.2d > 8.2.0a

It’s also possible to upgrade from a version earlier than 6.4.1b to 7.0.x or from 7.0.x to 7.2.x  – but this is a disruptive upgrade (meaning ports will go offline/online during upgrade). 

Brocade now has a document that describes a process of determining the ‘ideal’ version of Fabric OS you should be running. It is called Brocade FOS Target Path.
There is also a section (Recommended Migration Paths to FOS ) in the release notes describing how to get to the release you’re reading notes for. In addition to these there are Upgrade Guides from Brocade, at least for newer Fabric OS ( 7.4.0 and 8.0.0).

There are newer releases being released every now and then, in several series at the same time. You can think of it as releasing updates for Windows XP and 7 at the same time.
For example in February 2011 6.4.1a and 6.2.2e were released by HP. You can see this on HP’s site if you look at the date next to the download. Quite often Fabric OS versions are not released by the OEMs at the same time, for example “Customer Notice of 7.1.0a release 25th of March 2013” HP released 7.1.0a before IBM.

Which is the recommended one? Usually it’s the latest one in the highest series that the switch supports. If you have storage from more than one vendor you may want to check with all and see if they all support the version you want to upgrade to. Vendors certify their equipment with different firmware versions. If you have a tape library, ask the vendor if they have a recommended / list of certified versions.

HP: HP B-series Connectivity stream (available in HP SPOCK).
Brocade: “Brocade FOS Target Path
Other: Contact them for their compatibility matrices, for example IBM, HDS, EMC, Fujitsu.
Brocade also has their own “Brocade Fabric OS 7.x Compatibility Matrix” which lists compatibility with other vendors.

You could in principle also say that (some blades in directors are excepted from these generalizations):

2G cannot upgrade to Fabric OS 6.x
4G and 8G can be on Fabric OS 6.x
All 4G except some 4/8 & 4/16 (that’s 200E) and HP’s P- and C-class 4G blade switches (4012 & 4024) can run 6.4.x
8G can run Fabric OS 6.4.x
8G and above can run Fabric OS 7.x
16G (Gen5) needs to be on Fabric OS 7.x or Fabric OS 8.x
32G (Gen6) needs Fabric OS 8.x

 

Do you want to use the latest one in each series? Probably.
Do check for published advisories and the release notes in the firmwares.
Some models or blades may work on 7.0.x and not on 7.1.x or vice versa.
Fabric OS 7.3.x supports all hardware that supports 7.2.x.
Basically you need to read the release notes for at least the version you are upgrading to, to confirm that it supports your switch.

Download firmware links:

If you go to downloads for HP’s 4/16 there is a link that also takes you to the older FOS firmwares. If you don’t click through it also only have the firmwares that this switch supports. So the latest on there at the moment is 6.2.2f.

On the link above you can also download HP’s branded NA (Network Advisor, previously known as DCFM – Data Center Fabric Manager), see notes about that below.

If you click on manuals on the left side you will also be able to download release notes and other guides and references.

7.x and 8.x. can be found in the IBM links.

5.0.x firmwares can also be found at http://ftp.hp.com/pub/softlib/software12/COL22074/co-86832-6/FOS-Drawer_Statement.htm

Firmware Upgrade Order

You also probably want to decide on an order to upgrade the firmware on the switches.
It’s possible to do it via DCFM (now called Network Advisor, used to be something else) one switch at a time or even in parallel. I’d advice against doing it in parallel. One at a time and one step at a time seems the most cautious one. It’s not too bad to run a SAN with switches in different firmwares. One idea is to have all switches of one model on the same firmware. If you need to upgrade in several steps, do one step at a time.

Also switches that are of higher importance like Principal Switch, Core Switches or Seed Switches for DCFM/NA. Should you start with these or perhaps start with another switch of less importance to make sure the upgrade goes smoothly?

With more recent firmwares (6.4 and 7.x) it’s possible to jump more than one hop – if you are ok with disruptions in the network. Nice if you need to upgrade switches that aren’t in production.

Release notes:

FOS  IBM_Link HP_Link

Brocade release notes in .pdf

5.2.3
5.3.1c
6.0.0c
6.1.2c
6.2.2g
6.3.2e
6.4.3h
7.0.2e
7.1.2b
7.2.1g
7.3.2 
7.4.2c 
8.0.2e
8.1.2d
8.2.0a

Notes from the release notes:

Upgrading from Fabric OS 5.0.x to 5.2.3 is supported
Upgrading from Fabric OS 5.1.x to 5.3.1a is supported, but upgrading from Fabric OS 5.0.x or a previous release directly to 5.3.1a is not.
Upgrading to Fabric OS 6.0.0b is only allowed from Fabric OS 5.3.x. (6.0.0c is a special upgrade version, only meant to be used in between firmware upgrades)
Upgrading to Fabric OS 6.1.2c is allowed only from Fabric OS 6.0.0b
Upgrading to Fabric OS 6.2.2f is allowed only from Fabric OS 6.1.0a or later.
Upgrading to Fabric OS 6.3.2e is allowed only from Fabric OS 6.2.0a or later.
Upgrading to Fabric OS 6.4.3f is allowed only from Fabric OS 6.3.x. You can upgrade non-disruptively from 6.2
Upgrading to Fabric OS 7.0.2 can be done non-disruptively from Fabric OS 6.4.1a or later.
Upgrading to Fabric OS 7.1.2 can be non-disruptively upgraded from 7.0.x and 7.1.x. With caveats: For example, any previously existing error log entries with FOS v7.1.0 will be permanently lost once upgraded to FOS v7.1.2.
Upgrading to Fabric OS 7.2.x can be done non-disruptively from 7.1.x. Disruptively from 7.0.x is supported.
Upgrading to Fabric OS 7.3.x can be done non-disruptively from 7.2.x. Disruptively from 7.1.x is supported (see the FOS_UpgradeGuide_v730.pdf and the Brocade Release notes).
Upgrading to Fabric OS 7.4.x can be done non-disruptively from 7.3.x. From 6.4.x with firmwarecleaninstall
Upgrading to Fabric OS 8.0.x can be done non-disruptively from any Brocade 16G (Gen 5) platform and all blades in the Supported blades table running any FOS v7.4 firmware. From 7.3.0 with “firmwaredownload -s”
Upgrading to Fabric OS 8.1.x can be done non-disruptively from Brocade platform running 8.0.2 or later. From 7.4.x disruptively with “firmwaredownload -s”.
Upgrading to Fabric OS 8.2.x can be done non-disruptively from Brocade platform running 8.1.0a or later. From 8.0.x disruptively with “firmwaredownload -s”.

 

About non-disruptively: This means you can go to 7.0.x  from earlier  than 6.4.1a but ports will go offline during the upgrade.
See the release notes or Upgrade Guides for more details.

 

DCFM: Data Centre Fabric Manager / BNA: Brocade Network Advisor .

From 6.2.2a release notes:

With the introduction of Fabric OS 6.1.1, certain features and functions were removed from Web Tools (resident in the firmware) and migrated to the DCFM management application. HP recommends that, before you upgrade to Fabric OS 6.1.1x or later, if DCFM is not running on your fabric, you review the Web Tools functionality moved to DCFM, page 29 in these release notes and take note of what has changed so you can assess the impact on your fabric.

Fabric OS 7.x cannot be managed by DCFM 10.4 or BNA 11.0. You need BNA 11.1.0, see the release notes for 7.x.

Brocade Network Advisor 12.4.0 or later is required to manage switches running FOS 7.4.0 or later.
Brocade Network Advisor 14.0.1 or later is required to manage switches running Fabric OS 8.0.1 or later

Updates to documents

Sometimes Brocade releases updates to the manuals without actually updating the manuals. On HP’s page you can find them as “Documentation Updates”, “Fabric OS Administrator’s Guide Update”.

Fabric Watch and MAPS with FOS v7.3

Users running Fabric Watch for switch monitoring in FOS v7.3 are advised to convert to MAPS monitoring before upgrading to FOS v7.4. If you don’t, Fabric Watch will stop working.

Also the APM have been replaced with Fabric/Flow Vision.

Interoperability

See the release notes of the firmware for the specifics. For example Fabric OS 8.0.2 cannot be in the same fabric as for example HP C-Class 4/12 FC switches (4024) and one must use Fibre Channel Routing.

Prepare

Download old Brocade Fabric OS Firmware.

Basically, you need to update in steps.

To get FOS 5.2.1b and 6.0.0c firmware: Contact OEM Vendor or Brocade. I’ve found that two vendors have the firmware available online for free: HP and IBM, see below:
For HP products: here and click ‘Obtain Software‘, it has from 5.2.3 to 6.3.0d (but you’d rather use 6.3.2e). Eventually the link points to http://ftp.hp.com/pub/softlib/software12/COL22074/co-86832-6/FOS-Drawer_Statement.htm – this link sometimes changes.
Link to IBM’s page for downloading FOS 6 firmwares. This has firmwares going back all the way to FOS 2.6, it even has Fabric OS 6.0.0c and 5.2.3. On the page they have listed release notes and a little further down there is a link called “Release 6 Firmware”.
Actually, if you click on ‘Release 6 Firmware’ you are taken to a page on brocade.com where you can find many different firmwares, including 5.x and 7.x
IBM also have a link about FOS 7.x. Same for FOS 8.x

Also note that some features does not exist/work on older Fabric OS. For example on Fabric OS 5.1.x DHCP and SCP may not work (which forces you to use static IP and ftp).

Equivalent Product Names

Page with the equivalent Brocade and HP product names.
Page with the model number as seen in switchshow and HP’s model and Brocade’s model. This is a good one.
Page for correlating IBM and Brocade product names.

Recommendations

HP recommend that you upgrade one fabric and one switch at a time.
Waiting a week or at least a couple of days after you upgrade the first fabric is a good idea – gives you time to see if anything went wrong, if you can fix it and if you can do anything different next time.
See HP SPOCK for more details in regards to compatibility and interop modes.
The HP B-series Connectivity Stream lists the recommended firmware and all the supported ones for each switch model. It also has a list of the supported SFPs. Find it by clicking on “Switches” in the left-hand navigation pane under the “Other Hardware” section. The Connectivity Stream is great and it is updated often so I will not link directly to it. You need an HP Passport to log on to HP SPOCK – it is free to create and you do not need a contract or product in warranty.

Other vendors have similar matrices. HP for example does not have a list stating which Fabric OS firmware is supported with which HP P6000 firmware. The idea is that you go with the general recommendation of Fabric OS firmware.

Do read the release notes for the firmware(s) you decide on: for example not all 4GB SAN-switches can run the 6.4.x FOS. The 8- and 16-port 4Gbps switches (Brocade 200E) do not run 6.4.x or 6.3.x.
Only 8Gb and 16Gb switches can run the 7.x.x FOS.
The release notes also have the fixes, enhancements, upgrade paths and supported switches.
Generally the Brocade versions of the release notes are more verbose when it comes to fixes, but if you have an HP branded it might be easier to use the HP one as that has the HP names of the products. Also it might be hard to find the Brocade release notes if you do not have a contract with Brocade. Other vendors (like IBM/Fujitsu/HDS) provide you with the Brocade version of the release notes. You can find the release notes from their support pages.

Do  consider updating OS patches, HBA drivers/firmware, management softwares and storage drivers/firmware. For example Qlogic had driver updates to their drivers that prevent HP blades from getting stuck in G_port after a reboot. Another for Qlogic FC cards was to not write a partition table on Dell servers at 2TB on the LUN (not so nice for > 2TB disks)..

Upgrading Tools

SANLoader is an unofficial HP tool to upgrade firmwares. With this you do not have to create an ftpserver etc. Contact HP Support, they may give this to you.
This is meant to be used when the other ways does not work, but it helps out a lot as you do not have to set up an FTP/SCP server.

Sanloader used to (winter 2010) not work well on Windows 7 and may not work flawlessly on the pre 6.x firmwares.

Other ways:

  • Set up a ftp/scp server and upgrade via the CLI (command line interface).
  • Use DCFM ( Data Center Fabric Manager – now called Network Advisory ) to upgrade firmware.
  • Firmware can also be upgraded through the web interface (click on switch admin and then on firmware download). You will still need an FTP/SCP server for this though. See the web tools admin guide page 73-74 (FOS 6.2.x but it hasn’t moved).

FileZilla is a free FTP-server that works well. There are many alternatives around. But unfortunately some don’t work sometimes (not 100% sure but probably combination of older FOS with older ftp client with FTP server that couldn’t handle that client) as listed in the comments thread in this post. FileZilla is however still on sourceforge so you may want to be careful about installing that – it might contain malware. Storing them on a Synology NAS works – thanks Henny!

For FTP clients:

  • /usr/bin/ftp in Ubuntu (also in Ubuntu on Windows)
  • WinSCP for a free opensource Windows alternative that does both ftp and SCP (and more).

For SCP any machine with Linux and sshd on should work. You can also get an scp-server running on Windows, OpenSSH would work. Both protocols are old, SCP is safer while FTP is sending data in clear text.

Personally I like doing this via the CLI. The Network Advisor way gives you the possibility to upgrade in parallel, but that’s also risky. If you use a Linux server to provide the firmwares via SCP don’t forget to let the switches in via firewall or tcp.wrapper ( /etc/hosts.allow ). If you do the upgrade via ftp – make sure that passive and active ftp both works.

How to access the SAN-switch

The most common way is to access the CLI of a Brocade switch by connect to the IP of it with an ssh- or telnet-client, PuTTy is the name of a free Windows client. Telnet is unsafe so do try to use the ssh at all costs. Windows 10 has Bash which is in my opinion much nicer to use than putty.
It’s also possible to access the switch CLI via a serial cable, however as the firmware files are several 100MB (approaching 1GB for 6.4.x) that’s not really viable when upgrading firmware. Hyperterminal is a free windows tool that comes by default in some Windows versions. You can also use PuTTy for serial access.
To access the web interface just point the web browser to . It requires Java. The Java version that’s supported is listed in the release notes of the Fabric OS.

Upgrade

Here on HP’s Support Forum are some more notes about v6.x. Basic steps:

Note: version 6 does not require to specify the exact folder location SWBDxx: it just needs the root containing “the install” file

1) Unpack the downloaded firmware in the FTP or SCP download directory
2) Start the FTP/SCP Server and allow access
3) Connect to the CLI of the switch via telnet or ssh
4) Type this in the CLI: firmwaredownload
5) Answer all questions: when it asks for File Name be sure to write /v6.4.1b, that is the folder under which you find all the SWBDxx folders. Failing to do so makes it impossible to download the firmware
6) Wait for reboot of the switch and reconnect, check the firmware version with the “version” command

More notes about the upgrade

CLI Command to start the update process is firmwaredownload – this starts the interactive version, it is possible to specify user, directory, host directly via the CLI. See the Command Reference Guide for details. There are reference guides for each major Fabric OS release.

Specifying Directory

Please use forward slashes when specifying directories.

For example when you unzip the firmware file and it creates a sub-folder in the FTP-root that is called v5.3.1a then you need to specify /v5.3.1a as the directory.

For firmwares prior to 5.3.x you have to specify the release.plist – /v5.2.2a/release.plist.
However it says in the release notes for 5.2.3 that release.plist is no longer needed.

In some cases you may have to specify the sub directory.
For example the 4/16 HP Switch is a Brocade 200E with switchtype 34. So you would then use directory SWBD34 – /v5.3.1a/SWDB34. You can also try with /v5.3.1a/release.plist, /v5.3.1a/SWDB34/release.plist or /v5.3.1a/install. However with 5.3.1a you should not have to so /v5.3.1a should be enough.

firmwaredownload example:

switch:admin> firmwaredownload
Server Name or IP Address: IP.TO.SCP.SERVER
User Name: username
File Name: /path/to/v6.2.2e
Network Protocol(1-auto-select, 2-FTP, 3-SCP) [1]: 3
Password:
Server IP: IP.TO.SCP.SERVER, Protocol IPv4
Checking system settings for firmwaredownload...
System settings check passed.
You can run firmwaredownloadstatus to get the status
 of this command.
This command will cause a warm/non-disruptive boot on the switch,
 but will require that existing telnet, secure telnet or SSH sessions
 be restarted.
Do you want to continue [Y]: y
 Firmware is being downloaded to the switch. This step may take up to 30 minutes.
 Preparing for firmwaredownload...
 Start to install packages...
 dir ##################################################
 [[lots of these for all packets]] ##################################################
 [[also stuff like these are seen many times:]]
 warning: /etc/fabos/pki/switch.0.rootcrt created as /etc/fabos/pki/switch.0.rootcrt.rpmnew
 kernel-module-ipsec ##################################################
 Removing unneeded files, please wait ...
 Finished removing unneeded files.
All packages have been downloaded successfully.
 Firmware has been downloaded to the secondary partition of the switch.
 HA Rebooting ...

Transfer Protocol and Connectivity

If you are using SCP and that does not work, please try with FTP. If neither works, see if something else can log on to the FTP/SCP server. And of course, make sure the right permissions/root directory are set on the FTP-server. If your FTP/SCP server has log files, check them. If it works from one client but not from the switch, check the logs and see if there’s a difference. Sometimes if the SCP doesn’t work via CLI it might work by doing SCP (but starting it from the Web Tools, thanks Eric in the comments for this!).

If you are logged on as root on the SAN-switch you can use the scp- or ssh-client on the switch to confirm connectivity, like this:

ssh username@server ls /tmp/v6.0.1a to list the /tmp/v6.0.1a on the SCP server.

You need to be root to run the above command.

If that also does not work, you have some kind of networking problem – you can try direct connecting a laptop to the LAN interface of the switch. To see the network settings on the switch: ifmodeshow and ipaddrshow

Passwords

Sometimes when upgrading from 6.1.1d to 6.2.2 we have seen that the passwords have gotten reset.

Default password is then “password” or “fibranne”.

You can reset the password with the CLI command “passwd admin” to reset password on the admin account.

If you forget all passwords it might be possible to be able to reset it via the serial cable interface while booting the switch.

On EMC branded switches the default password might be: Serv4EMC

CF Cards

If your switch is out of warranty/contract and it’s still working. I’d suggest making a copy(dd  in linux for example) of the CF-card. Then if the CF card decides to fail you can just get a new one from random_electronic store and dd the contents of the flash back.

http://www-01.ibm.com/support/docview.wss?uid=ssg1S1003855

Licenses

When replacing a switch make sure that the licenses are correct.
If for example you have a switch with ‘power pack’ – then for HP there is a special spare part number for a switch with power pack and one without. Power pack is a grouping of licenses,  which licenses are in the pack differs between models.