Tag Archives: Storage

Red Hat – Clustering and Storage Management – Course Objectives – part 2

Post 1 – http://www.guldmyr.com/blog/red-hat-clustering-and-storage-management-course-objectives/ Where I checked out udev, multipathing, iscsi, LVM and xfs.

This post is about getting using luci/ricci to get a Red Hat cluster working, but not on a RHEL machine because sadly I do not have one available for practice purposes. So CentOS64 it is. Using openstack for virtualization.

Topology: Four hosts on all three networks, -a, -b and internal. Three cluster nodes and one management node.

Get the basic cluster going:

  • image four identical nodes
  • ssh-key is distributed
  • /etc/hosts file has all hosts, IPs and networks
    • network interfaces are configured –
    • set a gateway in /etc/sysconfig/network
  • firewall
    • all traffic allowed from -a and -b networks
    • at a minimum allow traffic from the network that the hostname corresponds to that you enter in luci
  • dns (PEERDNS=no is good with several dhcp interfaces)
  • timesync with ntpd
  • luci installed on mgmt-node # ricci is a web gui
  • ricci installed on all cluster nodes # this is the service talks with corosync
    • password set for user ricci on cluster nodes
  • create cluster in luci
    • multicast perhaps doesn’t work so well in openstack ?
    • on cluster nodes this runs “yum -y install cman rgmanager lvm2-cluster sg3_utils gfs2-utils” if shared storage is selected, probably less if not.
  • fencing is really important, how to do it in openstack would require a bit of work though. Not as easy as with kvm/xvm to send a destroy domain message.


  • Update and distribute cluster.conf
  • Have a service run on a node on the cluster (doesn’t have to have a shared storage for this).
  • Commands:
    • clustat
    • cman_tool
    • rg_test test /etc/cluster/cluster.conf start service name-of-service
    • ccs_config_validate


Share an iSCSI target between all nodes:

  • Using management node to share the iSCSI LUN.
  • tgtd, multipath
  • clvmd running on all nodes
  • lvmconf – make sure locking is set correctly
  • create vg with clustering
  • partprobe; multipath -r # do this often
  • vgs/lvs and make sure all nodes see the clusterd lv
  • minimum GFS filesystem is around 128M – you didn’t use all the vg right? =)
    • for testing/small cluster lowering the journal size is goodness
  • mount!


Red Hat – Clustering and Storage Management – Course Objectives

Attending “Red Hat Enterprise Clustering and Storage Management” in August. Quite a few of these technologies I haven’t touched upon before so probably best to go through them before the course.

Initially I wonder how many of these are Red Hat specific, or how many of these I can accomplish by using the free clones such as CentOS or Scientific Linux. We’ll see :) At least a lot of Red Hat’s guides will include their Storage Server.

I used the course content summary as a template for this post, my notes are made within them.. below.

For future questions and trolls: this is not a how-to for lazy people who just want to copy and paste. There are plenty of other sites for that. This is just the basics and it might have some pointers so that I know which are the basic steps and names/commands for each task. That way I hope it’s possible to figure out how to use the commands and such by RTFM.



Course content summary :

Clusters and storage

Get an overview of storage and cluster technologies.

ISCSI configuration

Set up and manage iSCSI.

Step 1: Setup a server that can present iSCSI LUNs. A target.

  1. CentOS 6.4 – minimal. Set up basic stuff like networking, user account, yum update, ntp/time sync then make a clone of the VM.
  2. Install some useful software like: yum install ntp parted man
  3. Add a new disk to the VM

Step 2: Make nodes for the cluster.

  1. yum install iscsi-initiator-utils

Step 3: Setup an iSCSI target on the iSCSI server.


  1. yum install scsi-target-utils
  2. allow port 3260
  3. edit /etc/tgt/target.conf
  4. if you do comment out the ip range and authentication it’s free-for-all


Step 4: Login to the target from at least two nodes by running ‘iscsiadm’ commands.

Next step would be to put an appropriate file system on the LUN.


Learn basic manipulation and creation of udev rules.

http://www.reactivated.net/writing_udev_rules.html is an old link but just change the commands to “udevadm” instead of “udev*” and at least the sections I read worked the same.

udevadm info -a -n /dev/sdb

Above command helps you find properties which you can build rules from. Only use properties from one parent.

I have a USB key that I can pass through to my VM in VirtualBox, without any modifications it pops up as /dev/sdc.

By looking in the output of the above command I can create /etc/udev/rules.d/10-usb.rules that contains:

SUBSYSTEMS=="usb", ATTRS{serial}=="001CC0EC3450BB40E71401C9", NAME="my_usb_disk"

After “removing” the USB disk from the VM and adding it again the disk (and also all partitions!) will be called /dev/my_usb_disk. This is bad.

By using SYMLINK+=”my_usb_disk” instead of NAME=”my_usb_disk” all the /dev/sdc devices are kept and /dev/my_usb_disk points to /dev/sdc5. And on next boot it pointed to sdc6 (and before that sg3 and sdc7..). This is also bad.

To make one specific partition with a specific size be symlinked to /dev/my_usb_disk I could set this rule:

SUBSYSTEM=="block", ATTR{partition}=="5", ATTR{size}=="1933312", SYMLINK+="my_usb_disk"

You could do:

KERNEL=="sd*" SUBSYSTEM=="block", ATTR{partition}=="5", ATTR{size}=="1933312", SYMLINK+="my_usb_disk%n"

Which will create /dev/my_usb_disk5 !

This would perhaps be acceptable, but if you ever want to re-partition the disk then you’d have to change the udev rules accordingly.

If you want to create symlinks for each partition (based on it being a usb, a disk and have the USB with specified serial number):

SUBSYSTEMS=="usb", KERNEL=="sd*", ATTRS{serial}=="001CC0EC3450BB40E71401C9", SYMLINK+="my_usb_disk%n"

These things can be useful if you have several USB disks but you always want the disk to be called /dev/my_usb_disk and not sometimes /dev/sdb and sometimes /dev/sdc.

For testing one can use “udevadm test /sys/class/block/sdc”


Combine multiple paths to SAN devices into one fault-tolerant virtual device.

Ah, this one I’ve been in touch with before with fibrechannel, it also works with iSCSI.
Multipath is the command and be wary of devices/multipaths vs default settings.
Multipathd can be used in case there are actually multiple paths to a LUN (the target is perhaps available on two IP addresses/networks) but it can also be used to set a user_friendly name to a disk, based on its wwid.

Some good commands:

service multipathd status
yum provides */multipath.conf # device-mapper-multipath is the package. 
multipath -ll

Copy in default multipath.conf to /etc; reload and hit multipath -ll to see what it does.
After that the Fun begins!


Red Hat high-availability overview

Learn the architecture and component technologies in the Red Hat® High Availability Add-On.



Understand quorum and quorum calculations.


Understand Fencing and fencing configuration.

Resources and resource groups

Understand rgmanager and the configuration of resources and resource groups.


Advanced resource management

Understand resource dependencies and complex resources.

Two-node cluster issues

Understand the use and limitations of 2-node clusters.


LVM management

Review LVM commands and Clustered LVM (clvm).

Create Normal LVM and make a snapshot:

Tutonics has a good “ubuntu” guide for LVMs, but at least the snapshot part works the same.

  1. yum install lvm2
  2. parted /dev/vda # create two primary large physical partitions. With a CentOS64 VM in openstack I had to reboot after this step.
  3. pvcreate /dev/vda3 pvcreate /dev/vda4
  4. vgcreate VG1 /dev/vda3 /dev/vda4
  5. lvcreate -L 1G VG1 # create a smaller logical volume (to give room for snapshot volume)
  6. mkfs.ext4 /dev/VG1/
  7. mount /dev/VG1/lvol0 /mnt
  8. date >> /mnt/datehere
  9. lvcreate -L 1G -s -n snap_lvol0 /dev/VG1/lvol0
  10. date >> /mnt/datehere
  11. mkdir /snapmount
  12. mount /dev/VG1/snap_lvol0 /snapmount # mount the snapshot :)
  13. diff /snapmount/datehere /mnt/datehere

Revert a Logival Volume to the state of the snapshot:

  1. umount /mnt /snapmount
  2. lvconvert –merge /dev/VG1/snap_lvol0 # this also removes the snapshot under /dev/VG1/
  3. mount /mnt
  4. cat /mnt/datehere


Explore the Features of the XFS® file system and tools required for creating, maintaining, and troubleshooting.


yum provides */mkfs.xfs

yum install quota

XFS Quotas:

mount with uquota for user quotas, mount with uqnoenforce for soft quotas.
use xfs_quota -x to set quotas
help limit

To illustrate the quotas: set a limit for user “user”:

xfs -x -c "limit bsoft=100m bhard=110m user"

Then create two 50M files. While writing the 3rd file the cp command will halt when it is at the hard limit:

[user@rhce3 home]$ cp 50M 50M_2
cp: writing `50M_2': Disk quota exceeded
[user@rhce3 home]$ ls -l
total 112636
-rw-rw-r-- 1 user user 52428800 Aug 15 09:29 50M
-rw-rw-r-- 1 user user 52428800 Aug 15 09:29 50M_1
-rw-rw-r-- 1 user user 10477568 Aug 15 09:29 50M_2

Red Hat Storage

Work with Gluster to create and maintain a scale-out storage solution.


Updates to the Red Hat Enterprise Clustering and Storage Management course

Comprehensive review

Set up high-availability services and storage.

Command View P6000 EVA Simulator 10.0

Due to somewhat popular demand here’s another post detailing the steps for somewhat successfully installing HP P6000 Command View Simulator on Windows 7 x64. It can be a bitch.

The older post is from 2011 with CV 9.4, this one also has PA – performance advisory bundled.

  • Download: http://software.hp.com
  • Two files: EVA Simulator 10.0 (Z7550-00252_EvasimInstaller_100fr_v1.exe) and a readme
  • There is an e-mail listed in the readme!
    • But if you want to, you can put in a comment below saying how sexy I am :p
  • The readme is quite long but most of it is about how to use the PA (performance analyzer), Appendix B is a required read. It describes how to add the Groups so you can log on to CV.
    • A previous blog post by myself truly also goes through how to add a user group :)

For lazy hounds:

  1. (optional) Disable UAC in Windows and make yourself admin.
  2. Put an account in the Windows Group called “HP Storage Admins”.
  3. Launch the downloaded file (it extracts a setup.exe and .msi file)
  4. Launch setup.exe – it’s located in the same directory where you launched the Z7550-00252_EvasimInstaller_100fr_v1.exe
  5. Next, next, next, next, yes, yes, Wait, yes, Installed!
  6. Try out the “Start HP P6000 EVA Simulator” new icon on your desktop, does it work? Profit!

“XF application has stopped working” – some friendly error I got and CV simulator did not start.. Most likely permission issue. Peaking through one of the command-prompts it repeats access denied.

It’s amazing that the CV simulator still relies on .bat scripts. Guess it’s for backwards compatibility with XP and Vista? Only one file necessary for all those Windows OS variants.

With default Windows security, the Simulator runs into a problem when it tries to write to files under c:\program files (x86)\ . There are probably many ways to remedy that, one might be step 1 above. This worked:

  1. Go to C:\Program Files (x86)\Hewlett-Packard\HP P6000 EVA Simulator\evasim
  2. Right-click on ‘start_bundle.bat’ and run it as an administrator. This should start the simulator.
  3. Open up a command prompt with Admin Privileges, cd your way into evasim directory and type: “start startcv.bat”
  4. That should launch the Command View process and also IE pointing to CV.
  5. If not, point your web-browser to: https://localhost:2374/SPoG/ or https://localhost:2374/
  6. Log in with the user/password you added into the “HP Storage Admins” group earlier.

Some tips:

In one of the “DOS” windows, there might be more clues as to what’s going on.

Open a command prompt with admin privilieges by typing “cmd” in the search bar then right-clicking and starting as administrator.

Inside the Simulator DOS prompt you can hit enter and if you see some commands (save, stop, exit, start) then that’s the simulator window.

If you want your changes to be kept, type “save” in the simulator window before quitting.

Some thoughts:

It feels a bit ruggish. I bet this whole mess could be improved quite easily with some decent scripts. Here’s one I’d like to see:

if $os == Win7:
    if $write_read_permissions_in_program_files != "allowed":
        print_in_big_letter("You need more axx! Do $THIS")


Setup a 3 Node Lustre Filesystem


Lustre is a filesystem often used by clusters because many computers can mount the filesystem simultaneously.

This is a small log/instruction for how to setup Lustre in 3 virtualized machines (one metadata server, one object storage server and one client).

Basic components:

VMWare Workstation
3 x CentOS 6.3 VMs.
Latest Lustre from Whamcloud

To use Lustre your kernel needs to support it. There’s a special one for server and one for the client. Some packages are needed on both.

Besides lustre you’ll need an updated version of e2fsprogs as well (because the version that comes from RHEL6.3 does not support large partitions).

Starting with the MDS. When the basic OS setup is done will make a copy of that to use for OSS and Client.

Setup basic services.

Install an MDS

This will run the MDT – the metadata target.

2GB RAM, 10GB disk, bridged networking, 500GB for /boot, rest for / (watch out, it may create a really large swap). Minimal install. Setup OS networking (static ip for servers, start on boot, open port 988 in firewall, possibly some for outgoing if you decide to restrain that too), run yum update and setup ntp. Download latest lustre and e2fsprogs to /root/lustre-client, lustre-server and e2fsprogs appropriately (x86_64). Lustre also does not support selinux, so disable that (works fine with it in enforcing until time to create mds/mdt, also fine with permissive until it’s time to mount).
Put all hostnames into /etc/hosts.
Poweroff and make two full clones.
Set hostname.

Install an OSS

This will contain the OST (object storage target). This is where the data will be stored.

Networking may not work (maybe device name changed to eth1 or eth2).
You may want to change this afterwards to get the interface back to be called (eth0). A blog post about doing that.

Install a client

This will access and use the filesystem.

Clone of the OSS before installing any lustre services or kernels.

Install Lustre

Before you do this it may be wise to take a snapshot of each server. In case you screw the VM up you can then go back :)

Starting with the MDS.

Installing e2fsprogs, kernel and lustre-modules.

Skipping debuginfo and devel packages, installing all the rest.

yum localinstall \ 
kernel-2.6.32-220.4.2.el6_lustre.x86_64.rpm kernel-firmware-2.6.32-220.4.2.el6_lustre.x86_64.rpm \
kernel-headers-2.6.32-220.4.2.el6_lustre.x86_64.rpm \
lustre-2.2.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64.rpm \ 
lustre-ldiskfs-3.3.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64.rpm \

The above was not the order they were installed. Yum changed the order so that for example kernel-headers was last.

yum localinstall e2fsprogs-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-debuginfo-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-devel-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-libs-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-static-1.42.3.wc3-7.el6.x86_64.rpm \
libcom_err-1.42.3.wc3-7.el6.x86_64.rpm \
libcom_err-devel-1.42.3.wc3-7.el6.x86_64.rpm \
libss-1.42.3.wc3-7.el6.x86_64.rpm \

After boot, confirm that you have lustre kernel installed by typing:

uname -av


mkfs.lustre --help

to see if you have that and

rpm -qa 'e2fs*'

to see if that was installed properly too.

By the way, you probably want to run this to exclude automatic yum kernel updates:

echo "exclude=kernel*" >> /etc/yum.conf

After install and reboot into new kernel it’s time to modprobe lustre, start creating MDT, OST and then mount things!
But hold on to your horses, first we ned to install the client :)


And then the Client

Install the e2fsprogs*

We cannot just install the lustre-client packages, because we run a different kernel than the ones that whamcloud have compiled the lustre-client against.

We can either back-pedal and install an older kernel. Or we can build (from source / SRPMS) a lustre-client that works on a kernel of our choosing. The later option seems like a better way, because we can then upgrade the kernel if we want to.


Build custom linux-client rpms

Because of a bug it appears that some ext4 source packages are needed – while they are not. You need to add some parameters to ./configure. This will be the topic of a future post.

The above rpmbuild should create rpms for the running kernel. If you want to create rpms for a non-running kernel you are supposed to be able to run.

Configure Lustre

Whamcloud have good instructions. Don’t be afraid to check out their wiki or use google.

/var/log/messages is the place to look for more detailed errors.

On the MDS

Because we do not have infiniband you want to change the parameters slightly for lnet to include tcp(eth0). These changes are not reflected until reboot (quite possibly something else) – but just editing a file under /etc/modprobe.d/ called for example lustre.conf is not enough.

Added a 5GB disk to the mds.

fdisk -cu /dev/sdb; n, p, 1, (first-last)

modprobe lustre lnet

mkfs.lustre –mdt –mgs


On the OSS

Also add the parameters into modprobe.

mkfs.lustre –ost


On the client

Add things into modprobe.


Write something.

Then hit: lfs df -h

To see usage!


Get it all working on boot

You want to start the MDS, then the OSS and last the client.
But while it’s running you can restart any node and eventually it will start working again.

Fstab on the client:
ip@tcp:/fsname /mnt lustre defaults,_netdev 0 0

Fstab on the OSS and MDS:
/dev/sdb1 /mnt/MDS lustre defaults,_netdev 0 0

While it’s running you can restart any node and eventually it will start working again.

Brocade Accredited Server Specialist – BASP


I’m currently preparing for yet another accreditation, the previous one I took was the BADCS,

The BASP (Brocade Accredited Server Specialist) appear to focus on the server side. Things like:

  • how to install drivers
  • HBA management tools
  • describe features
  • how to run diagnostics

This accrediation has the most questions of all the current ones, but it has the same amount of time allotted (one hour) so this exam will have a lot less time available for each question.

The curriculum for this accreditation are also free, they are called Introduction to HBA and Introduction to CNA. There’s also some docs about the 1860 Fibre Adapter. They can be found on Brocade’s Saba/training website under my.brocade.com.


// Update 20140422: This accrediation has been replaced with something else. See the current list here: http://www.brocade.com/education/certification-accreditation/index.page?

Brocade Accredited Data Center Specialist – BADCS


Time to study for another one :) Working my way towards the “Data Center Track”. To complete it it would be enough for me to complete 5 accreditations.

This one has a pretty cool name – BADCS!


I haven’t tried one of these Accredited exams before, but as far as I can tell:

  • Cheap: only 20$ USD
  • The exam is web based, no need to find a test center, you can do it exactly when you want to.
  • Accreditations do not expire
  • You don’t _have_ to take the course in the prerequisites before taking the exam, but it is recommended :)

Also, for this Accreditation the pre-requisite is the FC-101 course on brocade’s SABA page – and it’s free!

– The BADCS exam consists of 38 questions and lasts 60 minutes
– To pass this exam you must get a score of 71% or better 

So that’s about 27 correct out of 38 questions.

The objectives are on this page.

The only part I was initially not entirely sure about is the “Given a scenario, describe when portlog dumps are required”. The objectives indicate that a Fibre Channel theory knowledge is necessary, so the FC-101 course seems like a very good idea to study. I doubt many people remember specific FC mechanisms/theory if they don’t work with these occasionally. Like the well-known addresses – who remembers the address of the name-server or controller? =)

My general tip for the BADCS: Learn the material of the FC-101 course. Really. Learn. it.

You may be tricked into thinking that Brocade’s accrediations are easy because you can do them from home.

Access Gateway – NPV – TR

Say what??

Access Gateway – Brocade

NPV (N_port Virtualization (not NPIV) – Cisco

Transparent Mode – QLogic

These are all names for the basic idea / functionality but as there’s no standard the vendors have made up their own names for it.

A switch in Access Gateway (AG) mode does not consume Domain IDs, you can do port mapping, needs NPIV on the port in the switch that it connects to. AG requires a switch / fabric to connect to as it doesn’t run the normal fibre channel services.

It is very useful in case you are going to mix vendors in your fabric. Meaning you can populate the core with Brocade switches and then connect other vendors’ switches in the above modes to the Brocade switches.

On some QLogic switches you can also set a port into TR-mode, see this post on HP’s EBC forum about how to do it. It is not exactly the same as AG or NPV, because you still need to do zoning on the QLogic switch.

There is also the IPM by Qlogic for IBM – it looks like a module that you cannot switch between ‘fabric’ and ‘IPM’ mode. Which is what you can do on a Cisco or on a Brocade switch.


owncloud in a box!



Based on opensuse 12.1, I tested this out in vmware workstation by downloading the vm version (has a .vmdk file) inside the .tar inside the .tar.gz.

“Run the appliance, go to http://[yourservername]/owncloud and login as admin with the password owncloud42. Change the password in the admin interface and use the URL, which is shown there to access ownCloud via WebDAV.”

files, music, bookmarks, calendar, e-mail client, ldap and unix pam integration.

All in all, it looks like it could be quite cool. And it appears to be updated quite often too with a quite active community. But this takes a bit of commitment. First you want to move your existing stuff into this one (so you want to have backups of it etc) and you want to host it somewhere reliable (in case you access it through your phone etc). But I can see the potential, is quite nice to not have to depend on google or your other_random_”cloud”_vendor for these kind of services.

To create a new user group in Windows 7

This post is created upon request by a reader.

May or may not be needed for the P6000 Simulator. It is however required when you install the real HP P6000 Command View.

First you need to get into ‘Computer Management’, do this by right-clicking on ‘My Computer’. Then click your way into ‘Local Users and Groups’, and then into the ‘Groups’ section. In there, right-click somewhere and create a new group called ‘HP Storage Admins’ (or HP Storage Users for read-only). While creating it you can add a user (the one you log in with probably), you can also add it later by right-clicking the group.

Brocade Certified Fabric Designer – BCFD – Exam

Just took the BCFD (brocade certified fabric designer) exam two days ago.

Some tips:

Bring some water and food.

Good exam, but I am really tired of exams and certifications for now!

Also, isn’t easy to confuse Brocade Certified Fabric Designer with somebody who makes clothes?

Check out my other posts on the BCFD subject:


Brocade Certification – BCFD – Objectives


Data Collection

  • Given a scenario, design a solution that meets the customer’s requirements
  • Given a scenario, demonstrate knowledge of resiliency, redundancy, HA, and locality
  • Given a scenario, describe the various documents required in the design assessment

Practice by making up many scenarios and then deciding which is the best way to design it.

Management and Monitoring Tools

  • Given a scenario, describe how to satisfy a specific monitoring requirement
  • Demonstrate knowledge of Brocade management tools

What to monitor
How to monitor these

Hardware/Software Products and Features

  • Demonstrate knowledge of interoperability of B-Series/M-Series products
  • Given a scenario, describe Brocade hardware products and their purpose
  • Given a scenario, demonstrate knowledge of Brocade software features and purpose

Features: VF, FCR, TI, QoS, FW, IRL, Trunking, Port Fencing, D_Port

Distance Solutions

  • Given availability, performance and distance requirements, design an appropriate long distance solution using Fibre Channel
  • Given a specific set of requirements, demonstrate ability to design a SAN extension solution using FCIP

FastWrite, Tape Pipelining, SACK
Max distance for LWL and ELWL:
Max performance of FCIP:

Performance Tuning Optimization

  • Given a performance scenario, determine an appropriate solution
  • Describe strategies for maximizing throughput in a Data Center Fabric

ICL, nohops, trunking.
How to increase performance in FCIP and FCR:

Migration and Integration

  • Given an existing fabric, identify migration strategies to upgrade the fabric with new technology
  • Given a set of existing fabrics and network devices, determine a consolidation plan that minimizes disruption
  • Describe the requirements to integrate a Brocade DCX Backbone into an existing M-Series fabric

FCR, Integrated Routing, E_port on a switch in the M_series.


  • Identify requirements for restricting which switches/devices may join a fabric
  • Identify security features to restrict administrative access to a switch


BCFD – SAN Design Best Practices

This is a post in series of me studying for the BCFD – Brocade Certified Fabric Designer and it’s my comments on the document SAN Design Best Practices. Apparently this document is planned to be updated. The one I have is version 2.1. To find the latest go to My.brocade.com , documentation, Best Practices Guides. There’s also a “SAN Migration” guide there, but it’s from 2003 so irrelevant when it comes to anything specific, but ideas and reasons and methods might be valuable.

OK. I thought about doing something similar for this document as for the previous ones. But I just don’t feel like that, it’s basically just re-writing things in different wording so that it sticks in my brain. No instead I’ll post the questions that popped into my brain while reading it.

For a starter, I printed this .pdf. OK it’s not so environmentally friendly but it’s nice to have a break. One thing though, it took me a lot longer to read this than the course modules for BCFD. The SAN Design Best Practices is a first class pdf. At least in my opinion. I mean it’s general and specific. It needs to be general because there’s a lot of reasons behind designing things. Also, I don’t have any actual previous experience designing a SAN, so this is all new to me, and brings up a new side of Storage and Storage Networking that I just haven’t bothered much with before. Hopefully I have and will be learning a lot.


This paper refers to a lot of documents.

The “Brocade Scalability Guidelines” is not updated with 16G products (Only goes to FOS 6.3.0).


Page 10 it says “hop count is not a concern if the total switching latency is less than the disk I/O timeout value”.

Every switch hop adds latency (frame needs to be put in ASIC, processed then sent on its way).

Switch latency is measured in microseconds.
Disk I/O – is that the same as multipathing timeout? So 60 seconds for MPIO default in Windows?

How are these latencies measured?

Redundancy Resiliency

Two fairly similar words. One indicates something has a replica or a duplicate to fall back on. The other indicates the strength, can it by itself handle a problem.

Core switches should be equal or higher perf compared to edge switches.
Highest performing switch should be the principal switch.
Redundant links should be placed on different blades/ASICs or at least different port groups.

EHT – edge hold time

New timeout value that can discard blocked frames earlier than the 500ms default (down to 100ms). An I/O retry will still happen for each dropped frame.

Is a new features in FOS 7(confirm) and it is ASIC dependent. Meaning ports in another port group are not affected by the EHT in another port group.

EHT applies to all F_Ports on a switch and all the E_ports that share ASIC with F_Ports.

Intended for initiators only.


Directors interconnected via ICL is not considered a hop in FICON, is it in Open Systems?
Are the links uni-directional?

ICL cables should all have the same length.
ISL can be a bit different, max 30m in difference.
Don’t have ISL and ICL to the same switch/domain.


Hyper-Scale Fabrics: Scale-out Architecture with Brocade DCX 8510 Feature Brief.

Small Fabrics

Page 15: Brocade recommends core-edge as primary SAN design, or mesh for small fabrics (under 2000). !!! That’s pretty big..
On page 16 it says use full-mesh under 1500 ports.

Fan-In and Fan-Out and Oversubscription

Host ports to Target Ports

Device to ISL

Fan In : number of device ports that need to share a single port, be it target or ISL.

Consider: port queue depth, iops and throughput.

Example: If you have 4 devices with one 8G FC port each (32Gbps) and they are connecting over an ISL of 2x8G to another switch to a storage array that also have 2x8G then there is a 2:1 oversubscription, both on the ISL and on the target ports.

Bottleneck Detection

BD consumes switch memory, don’t monitor more than 100 ports on a 48k (no limit on DCX).

Start monitoring a small number of storage ports.

Fabric Watch

Thresholds and actions are generally different between initiators and targets. Thus place these on different switches.

FW Administrator’s Guide 7.0.0

Monitor Class 3 frame discards (C3TX_TO), they are an indicator of high-latency devices.

Fabric Watch Classes

This is a wide grouping of similar devices.
For example, temperature is a part of the class Environment.

Long Distance

Buffer Allocation

Number of credits: 6+ ((link speed Gb/s * Distance in km) / frame size in KB)

On 8510 4K buffers are available per ASIC to drive 16Gbps to 500km at 2KB frame size. With credit linking, buffers can be borrowed from a neighboring ASIC to extend distance.
Details about ‘credit linking’? Not many hits about this on google.

You can connect DWDMs in pass-thru mode where the switch is providing all the buffering.


FCIP adds a small latency (35 micro seconds). This is without the underlying TCP/IP delays.

Use QoS to give FCIP traffic highest priority.
Use CAR (committed access rate) to limit other traffic.
Use ARL (adaptive rate limiting) and set the limit to the remaining bandwidth.

FCIP traffic believes it is the only one using the bandwidth it has available, other traffic will suffer if they if they are sharing.

Use rate limiting on the FCIP on the Brocade systems, don’t limit it on the IP network.


This is mentioned for when extending mainframe/FICON extension over FCIP.

MLX is a Brocade Router.


OC1 =~ 52Mbps or without overhead ~50Mbps
OC12 = 12*52 or about 622Mbps
OC48 = 48*52 or about 2488Mbps

OC12 is recommended for Compression Mode 3 (GZIP/software only)
OC48 is recommended for Compression Mode 2 (SW with HW assist)

Neither of those are recommended for synchronous replications. Mode 0 is recommended and that is HW only compression.

Gaussian or Normal Distribution


Have fun.


There’s quite a bit about new Virtualization Engines in this paper. It basically means a device that has other disk arrays behind it, and then this device presents disks to servers. The danger is told to be that the engine can send a lot of small control frames, using up the buffer credits without using all the available bandwidth.

APM and Fabric Watch can apparently be used to monitor for excessive levels of SCSI reservations. How? – No specific details found but it is apparently threshold configurable in  fabric watch.


Less domains equals to reduced:

  • inter-switch zone transfers
  • name server synchronizations
  • RSCN processing

Dynamic Fabric Provisioning (DFP)

Only on Brocade HBAs and 16G.

Dynamically provision switch-generated virtual WWN.

Can be user-generated as well.

WWN stays the same even after HBA replacement.

In practice this means you can zone, QoS even before the HBA is online and before you know what the WWN is of the new device.

Brocade Certification – BCFD – Fabric Designer – Preparation

BCFD exam is going into Beta testing in January as well!

This post will be updated as I move along through the different objectives / documents.

// Update 2012-01-15: Added the Knowledge Assessment Test.
// Update 2012-01-28: Went through each .pdf and updated some in here.

The link to the Brocade page where it tells you how to register and where to get the material: http://community.brocade.com/docs/DOC-2379

# Note: This link no longer works

When are these available?
On Thursday 12/01/2012 at 0728 EET it was not available.
On Thursay 12/01/2012 at 0803 EET it was available.

So, that would indicate that the time Pearson follows is GMT-6 or Central Time.

On top of that the only available dates for me was 23rd and 24th of January :(
Time to study!
// update, that was changed, it was a mistake so now I get some more time to study :)

Exam Study Resources (page numbers are document page numbers, NOT the PDF page)

As I see it, the importance of each document could be arranged like this:

  1. CFD 200 Modules 3-7
  2. SAN Design Best Practices
  3. FOS Administrator’s Guide
  4. The rest.

With 1/2 sharing the top spot. I haven’t gone through the modules yet but I presume they all complement each other.
The reason for them sharing the top spot is because for this Beta Exam, the CFD200 material is for 8Gbps (and it has quite a lot of details about the M-series McData switches, which the 16Gbps BCFD did not include).

There is also a Knowledge Assessment on my.brocade.com ‘education’ page.
It’s called “CFD 201 8 Gbit/sec BCFD Knowledge Assessment”. Again, this is for 8G so beware that some stuff may not be up to date if you are doing the Beta for BCFD 16G. But, the actual type of questions is something that is useful. It mentions EFCM or Fabric Manager some times (this is the previous names of DCFM or what’s now called Brocade Network Advisor).

There is a nutshell guide for BCFD, but this is from November 2008 making it possibly even more outdated than the CFD200 material. But, because most of the topics are still valid it would still work as a refresher, but you can’t use it for anything specific.

I am doubtful that the M-series will be included in the BCFD 16G exam but as it’s still in the objectives for the 8G it’s probably wise to not skip that part completely. For that 1.5 years (half 2009 and 2010) when I did SAN support I only had one call about a McData switch.

Exam Study Resources with my comments:

CFD 200 BCFD Design Course Modules 3-7

  • Obviously these are the most important. I’ll go through these at a later stage.

Brocade DCX 8510 Backbone Family Datasheet


  • Lots of details about the system specs.

SAN Design Best Practices


  • Pages 2,5-16,19-26,31,32-36,40-45,51-53,55,58-62,66,67,72

Fabric OS Administrators Guide v7.0


  • Pages 37,43,66-70,102,142,151,153,157,196,199,241,273-286,301,314,315,320,372,383,395-398,402-406,414,417,425,429,437,438-443,449,454-461,464,503,504
  • topics
    • 256-area addressing
    • WWN-based PID assignment
    • enabling/disabling a port and port decommissioning
    • gateway links, ICL,
    • RADIUS/LDAP authentication
    • fddcfg / DCC/SCC policies
    • device authentication
    • ipfilter
    • firmwaredownload
    • advanced zoning (regular, broadcast, frame redirection, lsan, qos, ti)
    • traffic isolation zoning (and VF considerations for TI zones)
    • bottleneck detection
    • in-flight encryption and compression (technologies, enabling/disabling)
    • licensing (enable 10GbE, 7800, QoS, FCIP Extension, FICON acceleration, etc, etc, etc)
    • advanced performance monitoring (top talker, frame monitor, end-to-end)
    • adaptive networking (ingress rate limiting)
    • QoS prioritization (SID/DID or CS_CTL – class specific control)
    • trunking (ISL, ICL, EX_Port, F_Port)
    • Long Distance (buffer credit allocation, max distance, credit recovery)
    • FC-FC Routing (support platforms)
    • interopability (FOS vs M-EOS)

Fabric OS Command Reference v7.0


  • Pages  302,695,716,721,957,
  • commands
    • fcrconfigure  /  fcredgeshow
    • portcfgspeed
    • portdportest
    • portfencing
      • Why is the test for “Invalid Word Transmission” called ITW?
      • Ah, on portThConfig it is called “Invalid Transmission Word”.
    • supportshow

Fabric OS FCIP Administrators Guide v7.0


  • Pages 1,6
  • topics
    • FCIP platforms and supported features
      • 7800, FX8-24 and FR4-18i
      • FCIP Trunking
      • Adaptive Rate Limiting
      • 10GbE
      • 8G FC Ports
      • Compression (LZ and Deflate)
      • Acceleration (FCIP Fastwrite, OSTP)
      • QoS
      • VLAN Tagging
      • FICON
      • IPSEC
      • VEX
      • IPv6
      • Jumo Frames
    • 7800 switch hardware overview
    • FX8-24 has support for all features above, except: Jumbo frames (only FR4-18i supports those), IPv6 addresses for FCIP tunnels or IPsec, or 3rd WAN optimization hardware (the other do support this pre FOS 7)


Monitoring and Diagnostic Testing in Today’s High Speed High Density Networks

  • Pages 2-4
  • topics
    • powerpoint presentation of four pages in total
    • fc cable lengths
    • measuring loss
    • embedded diagnostics (bottleneck detection, fabric watch, frame monitoring, port fencing)
      • fmmonitor is a CLI that you can use to set up frame monitoring, for example SCSI reservations and aborts.

Brocade Network Advisor SAN User Manual


  • Pages 12,164,186,255,596,770,794,796
  • topics
    • “Connectivity Map Toolbar” & “Product List”
    • Call Home Feature
    • Copying and Deleting Views
    • SAN Device Configuration (configuration repository management)
    • LSAN Zoning
    • Performance Overview
    • Bottleneck detection

Why dB Loss Matters for Building Reliable Stable Networks


  • Pages 2,3
  • topics
    • total 8 pages
    • link lengths and link loss budgets

Brocade 6505 Hardware Reference


  • Pages 13,15
  • topics
    • ISL trunking
    • switchstatuspoolicy
    • fos native and AG modes

Brocade Access Gateway Administrator’s Guide


  • Pages xiv,72,
  • topics
    • supported hardware and software (which switches and FOS)
    • enabling NPIV on M-EOS and Cisco switches
      • CISCO: config t; npiv enable
      • MEOS:
    • new features -F_Port static mapping, APM, B6510, Target Aggregation, Direct target attachment, N_Port monitoring

“You can run the agshow command to display Access Gateway information registered with the fabric. When an Access Gateway is exclusively connected to non-Fabric-OS-based switches, it will not show up in the agshow output on other Brocade switches in the fabric.”

CEE Admin Guide 53 1002163-02

  • Page xviii
  • topics
    • Supported Hardware: Standalone switch B8000 and the blade FCOE10-24
    • IGMP configuring (IGMP is used in multicast, ethernet)
    • Replacing the B8000
      • configdownload
      • and copy running config and stuff! Looks very similar to the Cisco CLI.

Brocade Adaptors Admin Guide


  • Pages 3,13,
  • topics
    • AnyIO technology on the 1860 Fabric Adapter, just change the SFP and set the mode with bcu port –mode or bcu adapter –mode.
      • HBA or FC mode (FC)
      • Ethernet or NIC mode (GbE)
      • CNA mode (FCoE)
    • Adapter Support (OS + description of adapters)

The New Data Center 1st Edition

ISBN: 978-1-4507-0195-2

  • Pages 65,66,78
  • topics
    • Fabric Based Disaster Recovery (64-67)
      • An overview of some of the extension technologies and reasons behind them.
    • Network Security (77) + Power, Space and Cooling Efficiency (78)
      • Network Security is not FC related.

RHCSA – Rapid Track – Day 4

Last day!

A little more kickstarting, LVM – logical volume management and File ACL.
Then a rehearse of the previous chapters.

Feeling a bit excited about tomorrow!

These ACL were a bit more complex than I thought, but they could be made very complicated if you want to. But there’s the default ACL and then there’s the normal ones. Chmod +s for sticky bits.

Also got a 4GB USB pen that does about 4.4MB/s :p

Some important ones:

mount -o remount,rw /
kernel-doc package and /usr/share/doc/kernel-*
rpm -qd; rpm -qc

Some important but not importantest:

getsebool -a
setsebool -P
usermod -a
sssd - service that caches authentication stuff

/etc/auto.master ->

/home/guests    /etc/auto.guests

/etc/auto.guests ->

* -rw nfsserver:/path/to/mount/on/home/guests/&

If on nfsserver there is an nfsshare that is: /path/to/mount/on/home/guests/
and in there you have home directories for users. Then this will automount these directories when anybody tries to access them. Same concept as doing:

ls /net/nameofnfsserver/

RHCSA – Rapid Track – Day 3

Wow, what a day!

Some great stuff today:

It ended with configuring a kickstart file, starting an unattended installation via PXE by referring to said kickstart file on an http-server.

Before that we were playing with partitioning, making swap, encrypting with cryptsetup and LUKS. Also very interesting.

After we learned about partitioning we did changing booting kernel parameters, resetting root password, editing grub, loading modules, setting module specific parameters etc.

And we started the day with SELinux. Very interesting, a lot of opinion about that in the room but honestly I can see that SELinux is extremely useful and doesn’t cause much harm on a desktop. Especially one where you don’t run any services. Also, there’s so often 0-day exploits for various net-services that running SELinux can’t be bad, right? There’s probably lots of other stuff you can do to do hardening in a Linux system.

Teacher did not mention chcon at all, only restorecon.

More thoughts

This RHCSA course do assume that you know a bit. For example it assumes that you know scripting, we’re not going through that at all. Using vi, less, are also assumed. Parsing, grepping etc. There are people struggling keeping up in class.

Going through all the objectives before attending is a great idea. It gives you some breathing room while doing the exercises in class and if you have stumbled upon and questions while you were experimenting yourself – you have a great opportunity to ask these in class. Another good thing with this is probably that it makes you faster at doing the task. If you can reset the root password on a VM in 60 seconds, instead of 300s – because you’re wondering about what commands to run, what parameters to send and how to send them, etc, that’ll save you a lot of time.

Mini Book Review

But even that is not enough, you really need to be experienced with Linux before. How to use the CLI and things like that. There are some good books around. Such as the book UNIX and Linux System Administration Handbook. I haven’t read all of it yet, actually just the part until Perl scripting, which is only the 2nd chapter! The stuff before chapter 3 are just basic linux administration / using the system. After that it goes into booting, filesystem, basically everything, and this is with some serious depth. Which is not for me.


Found http://www.youtube.com/watch?v=CjVYnK57YLA on youtube. Pretty cool, snapshotting in LVM!

BCFP 16G Beta – I passed!

In a moment of frustration/impatience I posted on ‘Brocade Certified’ that it was still 3 weeks until the results are posted. Good boss of training Joe Cannata replied that I could e-mail him and get the results before. I did, and I passed! Most excellent and kind of them to do that! So nice to not have to wait the extra three weeks until it’s entered/updated in the Pearson Vue system. What kind of name is that anyway? Pearson Vue? Latin? Turns out no.

VUE stands for: Virtual University Enterprises
Pearson is just from somebody’s name.

Maybe there’s another meaning behind the name Pearson (besides son of pear).

Back to the subject, how nice was it to find out that I passed? The first few days / week after the exam I was really aching to find out. Then I pretty quickly forgot about it. It wasn’t until maybe a month and a half later that I thought, cmooon why is this so slow! All in all it feels pretty awesome to have the certificate! It took a pretty heavy amount of studying. I think maybe now I got the courage to go for the designer one as well!

BCFP 16G : Post-Exam

OK, did the exam. Took 2.5 hours.

It felt like I was plowing through them pretty fast but after some 80-questions the speed dropped significantly and I started to have trouble focusing. Short unfocus/try to think about something else , stand up a bit helped. This test wasn’t as nerve-wrecking for me as the BCFA – maybe it had something to do with that I didn’t pay for this one.. (smålänningen i mig).

I realized last night that I hadn’t listened to the BCFP nutshell guide (the one with audio) so I quickly went through that and I believe it helped a bit, even though it’s for FOS 6.3 and 8G FC stuff. If you do this exam when it goes live, definitely do listen to the nutshell, maybe several times and don’t forget to click the ‘notes’ button on the slides, there’s more info there. It’s kind of a very sped up/condensed version of the whole material, but there’s some things that aren’t even mentioned that you’ll only find in the material. But it is free. It’s nice to have somebody read it out loud, maybe they phrase it a little different than you do (in the head) that is just what’s needed for you to understand it.

Also there are the simulations/labs in the material when you buy that’s really valuable.
For the BCFA they had virtual class rooms for this nutshell – looks like they might have changed this now and recorded it. One avenue less to ask questions.

Now the waiting begins. Two months until I get to know the results. Yikes!

Gotta find something else to occupy myself with now.. hmm..

Archiving? Buddhism? Apparently BCFD (fabric designer) is beta-tested in Dec/Jan. This would be the next logical step for me. Why not huh? =) Anybody has any ideas? Right now I just feel like getting myself into the next Malaz book by Ian C. Esslemont – Stormwielder, supposedly about Greymane!

BCFP – good threads in the brocade forum

How to set up two FCIP tunnels between two B7500: http://community.brocade.com/message/15261

Connecting two sites via FCR: http://community.brocade.com/message/16828#16828

Set up FCIP between two B7800: http://community.brocade.com/message/14216#14216

Why is there an IP in ipaddrshow on FC interfaces: http://community.brocade.com/message/3978#3978





BCFP – 16G Studying

Another studying tip!

Head over to the forum/community at Brocade. Do it. After you read this post :p

For example in the BCFP there will be questions about NPIV, so it makes sense to read the forum for threads regarding NPIV. Right? I mean there’s bound to be troubleshooting, and getting some ‘real’ experience troubleshooting SAN is quite hard to get, especially with stuff like FCIP/FCR.

Example link: http://community.brocade.com/message/18897#18897

You don’t need an account. If you register you don’t get access to much anyway.

Here is one as an example. But there are lots of posts there and there’s a ton to learn. I usually just troll/help out on HP’s Enterprise Server/Storage forum but I think I’ll start reading on this as well.

SAN Primer – Introduction to Data Storage

You may have heard about this storage or SAN stuff, but what is it? Is it complicated and cool? Yes. Now it doesn’t have to be complicated, but it sure can be sometimes.
This post is just a brief primer/introduction to storage and what it entails. In case maybe you got a job interview or just would like to know a little bit more about it.

I’ll update this post as I go, last update 2012-07-13 – added some books and free pdfs and links.

What is a SAN?

‘Storage Area Network’ – or storage network.
Generally it doesn’t have to be a ‘network’ it could just be direct connected equipment or peer 2 peer. But what it always entails is a shared storage, most often disk or tape.

What is in a SAN?
When it comes to disk storage on fibre channel there’s a few standard components: FC HBA in the server, SFP and cables, SAN-switch, SFP and cables, FC port in the disk array controller and then there’s something behind the controller that connects disks.

You can connect the FC HBA directly to the disk array.

What is storage?
It’s somewhere where you can store data. Most common today would be: hard drives, flash drives (ssd), magnetic media (tape) and optical media (dvd/blueray/cd).  In a computer you cannot fit hundred of hard drives, but sometimes there is an application that requires lots and lots of data (maybe for example CAD drawings, video editing). This is when a SAN comes in, with only the help of for example a fibre channel card you can give a server access to lots of storage.

How do you do it?
If you want to give a server disk space from a fibre channel SAN this is what you do:

  1. Fullfil the hardware requirements (so fibre channel HBA+drivers and multipath software, SAN-switch, disk array and sfps + cables)
  2. On the SAN-switch create a zone with the disk array’s and the FC HBA’s domain id, port id or port wwn. It’s possible to do it without zones, but they are good for fault isolation.
  3. On the disk array you should now see the server/host, create a disk and map/present it to the host.
  4. On the host you most likely need to do a rescan/reinitialize of the fc-bus.
  5. After the server sees the LUN it will have a new hard disk available, you can use your normal partitioning/format/filesystem tools to create some usable space.

Can I use the same disk on two servers?
This is a pretty common question, the answer is sometimes and the sometimes depends on which file system you are using. It needs to support that more than one host can access it at the same time. NTFS does not support this and if you try it anyway you’ll corrupt the file system. For Windows you need to look into CSV – clustered shared volumes or other networked file systems like NFS/CIFS.

What is the difference between fibre channel and iscsi?

FC is sending SCSI commands over fibre channel, it’s not always fibre or optical cables.
While iSCSI is sending SCSI commands over TCP/IP.
FC is a whole network technology while iSCSI is running on top of a network technology -> TCP/IP.

Some literature:

Both the IBM and the HP one are quite lengthy. The HP one has a lot of HP specific guides, best practices and supported configurations. The FC 101 by Brocade actually goes quite deep into the theory of the FC protocol.