BCEFP 2015 certified!

Passed the Brocade Certified Ethernet Fabric Professional 2015 exam in May and I finally got the results back!

https://www.certmetrics.com/brocade/public/transcript.aspx?transcript=8XRF1FE12MR41GC4

This one felt quite hairy compared to the other tests I’ve taken. Definitely recommend doing the course / getting some real hands-on experience for these certifications.

Lustre 2.5 + CentOS6 Test in OpenStack

Reason: Testing to Lustre 2.5 from a clean CentOS 6.5 install in an openstack.

Three VMs: two servers, one MDS, one OSS and one Client. CentOS65 on all. An open internal ethernet network for the lustre traffic (don’t forget firewalls). Yum updated to latest kernel. Two volumes presented to the lustreserver and lustreoss for MDT + OST, both are at /dev/vdc. Hostnames set. /etc/hosts updated with three IPs: lustreserver,  lustreoss and lustreclient.

With 2.6.32-431.17.1.el6.x86_64 there’s some issues at the moment for building the server components. One needs to use the latest branch for 2.5 so the instructions are https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821

Server side

MDT/OST: Install e2fsprogs and reboot after yum update (to run the latest kernel kernel).

yum localinstall all files from: http://downloads.whamcloud.com/public/e2fsprogs/1.42.9.wc1/el6/RPMS/x86_64/

Next is to rebuild lustre kernels to work with the kernel you are running and the one you have installed for next boot: https://wiki.hpdd.intel.com/display/PUB/Rebuilding+the+Lustre-client+rpms+for+a+new+kernel

RPMS are here: http://downloads.whamcloud.com/public/lustre/latest-feature-release/el6/server/SRPMS/

For rebuilding these are also needed:

yum -y install kernel-devel* kernel-debug* rpm-build make libselinux-devel gcc

basically:

  • git clone -b b2_5 git://git.whamcloud.com/fs/lustre-release.git
  • autogen
  • install kernel.src from redhat (puts tar.gz in /root/rpmbuild/SOURCES/)
  • if rpmbuilding as user build, then copy files from /root/rpmbuild into /home/build/rpmbuild..
  • rebuilding kernel requires quite a bit of hard disk space, as I only had 10G for / then I made symlinks under $HOME to the $HOME/kernel and $HOME/lustre-release

yum -y install expect and install the new kernel with lustre patches and the lustre and lustre modules.

Not important?: WARNING: /lib/modules/2.6.32-431.17.1.el6.x86_64/weak-updates/kernel/fs/lustre/fsfilt_ldiskfs.ko needs unknown symbol ldiskfs_free_blocks

/sbin/new-kernel-pkg –package kernel –mkinitrd –dracut –depmod –install 2.6.32.431.17.1.el6_lustre

chkconfig lustre on

edit /etc/modprobe.d/lustre.conf and add the lnet parameters

modprobe lnet
lctl network up
# lctl list_nids

creating MDT: mkfs.lustre –mdt –mgs –index=0 –fsname=wrk /dev/vdc1
mounting MDT: mkdir /mnt/MDT; mount.lustre /dev/vdc1 /mnt/MDT

creating OST: mkfs.lustre –ost –index=0 –fsname=wrk –mgsnode=lustreserver /dev/vdc1
mounting OST: mkdir /mnt/OST1; mount -t lustre /dev/vdc1 /mnt/OST1

Client Side

rpmbuild --rebuild --without servers

cd /root/rpmbuild/RPMS/x86_64
rpm -Uvh lustre-client*

add modprobe.d/lustre.conf
modprobe lnet
lctl network up
lctl list_nids

mount.lustre lustreserver@tcp:/wrk /wrk

lfs df!

BCEFP 2015 – Studying for the exam – part 3

This third post  focuses on the remaining sources of information I had for studying for the BCEPF. At the time this post is published I have taken the exam.

When I make comments to CLI commands I put them after a #.

This is part of a series of posts on the topic of studying for Brocade’s Certified Ethernet Fabric Professional.

The two previous posts: Objectives and reading materialscourse and nutshell guide and NOS Admin Guide

 

VDX Troubleshooting Course

 

The material available also feels very short, same as the beta material available for the CEF300 , like only the parts of the slides that were updated for the BCEFP 2015 beta were included.
When a slide says “(cont.)” but there was no previous slides on this topic, that’s a hint :)
Take the (currently free) course on Brocade’s SABA – it’s under Education on my.brocade.com. It has way more slides and info.

 

Some notes from the course:

Firmware Upgrade

  • Can upgrade all/selected RBridges in a logical chassis: firmware download logical-chassis
  • FTP/SCP/SFTP/USB(only local switch with USB)
  • By default it stages firmware only – so no reboot or activate. By adding auto-activate it reboots all RBRidges at the same time, not recommended.

SNMP

  • When BNA discovers a switch it automagically configures the switch to send traps (UDP 162) to the BNA server.

Fabric Formation:

  • Requires: Licenses. Same VCS ID, unique RBridge ID and same VCS mode (Fabric Cluster or Logical Chassis)
  • Check:
    • ISL ports are operational (show fabric islports)
    • Incompatible Firmware Levels

ISLs:

  • no fabric isl enable # this disables ISL formation. This makes it an edge port
  • CPU could be too busy to send ISL keepalives
  • If ISL is segmented and interface is up/up – it’s probably a config issue.

 

vLAGs:

  • show running-config interface TenGigabitEthernet 1/0/2 # shows config
    • no shutdown
    • channel-group $NUMBER mode active type standard # active – LACP. Standard/Brocade proprietary.
  • show interface TenGigabitEthernet 1/0/2 # shows status
    • When counters are non-zero and looking for errors. Clear them and compare the delta.

 

Other:

  • show interface stats brief # shows discards, errors and CRC
  • VRRP:
    • show vrrp detail
    • pre-empting : if a virtual router comes online with higher priority than the current it will take over
    • VRRPE: Can enable short-path-forwarding. If one of the backup virtual routers (that don’t own the Virtual IP) can actually forward traffic if that is advantageous.

 

FCoE:

  • show running-config zoning # show FCoE zoning
  • show fabric all #
    • RBRidge with this name: fcr_fd_160 # this comes online when fabrics are connected and Fibre Channel Routing is used.
    • RBRidge with this name: fcr_xd_4_100 # this comes online when devices across FC Fabrics can communicate. Don’t see this? Check zoning.

iSCSI:

BCEFP practice questions / answers

http://community.brocade.com/t5/Certification/BCEFP-2013-Exam-150-180-Practice-Questions/ta-p/4099

These are decent practice questions and is nice because the answers give some explanation to the answers too.

Other

Intro to VCS Fabric Technology: http://www.brocade.com/downloads/documents/white_papers/intro-vcs-fabric-technology-wp.pdf

CFP- MSA CFP2 Hardweare Specs:

  • about the 40/100Gbps CFP2 SFP. MSA – multi-source agreement.
  • CFP2 module shall support LC, MTP12 and MTP24 optical connector types. MPO

NOS 4.1.1 release notes (p4,10,28,50): 

  • 4.1.0 and later support VRRP-E across VCS fabrics.
  • 4.1.0 and later have vlag ignore split on by default
  • clear mac-address-table can clear MAC addresses associated with vLAGs and on other switches
  • Page 50 Has a table of scalability numbers for various features such as (6710 VCS, 6740 VCS, 8770 VCS):
    • max members of a LAG (8,16,8)
    • max switches in a fabric/logical cluster (24,32,32)
    • max ECMP paths (8,8,16)
    • max member ports in a vLAG (64)
    • max member of VMs (8k)
    • max ARP entries (8k,12,50k)

 

Network OS Command Reference v4.1.1 53-1003226-01

Pages 299, 1258-1260,1266,1297,1317,1318

  • firmware download
  • snmp-server user # access
  • snmp-server v3host # trap recipients
  • spanning-tree edgeport # quickly transitions to forwarding state: only for RSTP/MSTP. Portfast for STP.
  • switchport access # only allows untagged and priority tagged
  • switchport trunk allowed vlan ${rspan-vlan} # add allowed VLAN on trunks on L2 interfaces in trunk mode
  • switchport trunk default-vlan # put all non-matching traffic into this VLAN

 

Hardware reference manuals

VDX 6740 Hardware Reference Manual 53-1002829-02: Page 1

  • 6470: 24 1/10GbE SFP+ ports.
  • 6740T: 24 RJ-45
  • 6740-1G: 48 RJ-45 Base-T. 10Gb with license.

VDX 8770-4 / 8770-8 Hardware Reference Manual 53-1002563-03: 

  • Chapter 1, Page 1:
    • Features CloudPlex.
    • Requires NOS 3.0.0 or greater.
    • 8770-8:
      • Up to 384 10GbE or 96 40GbE. Dual MM. 6 SFM. Max 8 PSU. 4 Fans. SX or LX 1Gbps SFP transceivers.
    • 8770-4:
  • Chapter 3, Page 32
    • For copper connections to < 1Gbps BaseT switches a crossover cable is needed (but it might not be if MDI/MDIX works..).
    • LC connectors for fiber ports

VDX 6730 Hardware Reference Manual 53-1002389-06: Pages 1,2,15

  • 6730-32: 32-ports. 6730-76: 76 ports. 8 or 16 x 8GB FC ports.

 

Network OS Software Licensing Guide v4.1 53-1003164-01

Pages 11-13

  • All have FCoE license (except 6710).
  • All have POD licenses (except 8770)
  • 6740 have 10/40GbE port upgrades
  • 8770 have L3 and Advanced Services

Notes:

  • for multi-hop FCoE it is needed on each node
  • L3: OSFP, VRRP, PIM-SM, Route-Maps, prefix list
  • Advanced: FCoE and L3
  • After installing a time-based license you cannot change system date or time. NTP is however not blocked. If you are using NTP, don’t change system date/time when a time-based license is installed.

BCEFP 2015 – Studying for the exam – part 2

This second post  focuses on the NOS Admin Guide.

When I make comments to CLI commands I put them after a #.

This is part of a series of posts on the topic of studying for Brocade’s Certified Ethernet Fabric Professional.

The two previous posts: Objectives and reading materials and course and nutshell guide

The NOS 4.1.1 Admin Guide

I’ve been reading the pages on paper (together with a highlighter :) that I printed with the help of my script below and there is lots of goodness in there.
For sure some topics are brought up without any preamble so for these I just make a note in the paper that I need to check out this other thing later.
Especially the Fibre Channel things take up quite a lot of pages. I thought in these devices FC would not be with so much focus but it seems like they do re-use a lot of the things in FC that works.

Notes and acronyms (page in NOS Admin Guide):

  • DCB – lossless. Able to allocate bandwidth on links.
  • TRILL – transparent interconnections of lots of links.
  • RBridge – Routing Bridge. Lowest WWN or priority.
  • Looks like on p54 only the text about Logical Chassis cluster config is applicable.
  • Trunking between VDX8770 and B8000 are not supported (B8000 is some early version of FCoE from Brocade, not visible on Brocade’s page where they list their switches)
  • ECMP – Equal-cost multi-path routing (p149)
  • AG – VCS must be enabled for Access Gateway
  • AMPP – Automatic Migration of Port Profiles – some OK pictures around p375
  • VRF – Virtual Routing and Forwarding

 

Questions:

  • There is also a Openstack Neutron Plugin (p29)
  • Would be good to include also page 114 before page 115 to see what they mean with leaf/spine/core (p115)
  • OOB access to console is via serial (p115)
  • How to reload a group of switches? (p115)
    • reload system rbridge-id all
  • Does trill use IS-IS type link-state? (p136) Yes
  • Can VF_Ports be anywhere in the fabric? (p202) Yes, they must be mapped to N_Ports.
  • Is there no web interface on the VDX? (p269) Probably not, there are some “http server” and “ip http-server” commands.
  • What are valid upgrade paths? Not so clear. 3.0.0 to 4.0.0 is not OK. 3.0.1 to 4.0.0 is OK. (p341)
  • What is this netinstall? (p371) – 10 hits on google: brocade “netinstall” vdx
  • What does the asterisk mean in the output of “do show vcs” ? (p597) The one you are running the command on? Is not principal RBRidge, that is >.

 

Commands (# comments) (page):

  • backup config: copy rbridge-running-config rbridge-id rbridge-id location_config
    • copy rbridge-running-config rbridge-id 2 scp://user:pw@host
  • vcs
    • no vcs logical-chassis enable # remove a node from logical chassis cluster (p76)
    • vcs replace rbridge-id 3 # replace RBridge with id 3 (p77)
    • enable (p139)
    • virtual ip address 10.1.1.1 (p143)
  • config terminal # to enter global exec mode (p94)
  • firmware download (p119)
  • logical-chassis principal-switchover (p138)
    • and logical-chassis principal-priority are the only logical-chassis commands
  • disabling a port:
    • shutdown # on an ISL brings down link and FSPF adjacency.
    • no fabric isl enable #  link stays up, shorter reconvergence
  • show
    • vcs virtual-ip (p143)
  • vcenter/vnetwork # used to connect to a vcenter and to discover hosts. (p243)
  • bind # create persistent binding between logical FCoE port and 10G/40G/LAG port. Port or MAC, not both. (p345)
  • enable statistics direction # for VXLAN tunnels to enable statistics on VLANs. (p365)
  • no spanning-tree shutdown # default for all VLANs – meaning it’s enabled. (p381)
  • lacp system-priority 25000 # For deciding which system is in charge of resolving LAG conflicts. (p437)
  • nas server-ip IP/PREFIX # Set IPs for AutoQoS for NAS (p506)
  • address-family ipv4 unicast # Used to enter IPv4 config in a VRF (p609)
  • debug lacp pdu # turn on debug (p714)
    • terminal monitor # view debug messages in terminal

Printing the NOS Admin Guide relevant pages:

 

Because the slides for the BCEFP course were insufficient I would get a lot of the basic information about the NOS from the NOS Admin Guide.
In the materials provided the NOS Admin Guide was separated into two documents. The guide is of course available in one pdf. Go to the web version and click on the pdf icon.
This makes printing based on the numbers provided easier. However the NOS Admin Guide for v4.1.1 referenced was one version below the one on the html version.

Now the numbers referenced are the numbers in the document, not the one told by the pdf viewer. So actually page 11 is page 13. Page 135 is 137. 311 is 313. 425 is 427. 517 is 519. 661 is 663. 714 is 716.
I checked a few to make sure there were no major increase due to version difference or elsewhere. One could with a bit of scripting increase each number with two like:

1,13-22,28-33,56-58,77-79,96,117,121,137-146,151,152,193,203-205,212,245-249,255,263,271,313-316,323,324,340-347,363-387,402,405,408,427-435,439,467,485,497,506,508,519-523,543,561,565,567,585,595,596,599,605-611,663-665,670,678,684,688,716,717

Cover page added to make it look nicer when printing. Old numbers:

Network OS Administrator’s Guide v4.1.1 53-1003225-01

Pages 11-20,26-31,54-56,75-77,94,115,119,135-144,149,150,191,201-203,210,243-247,253,261,269,311-314,321,322,338-345,361-385,400,403,406,425-433,437,465,483,495,504,506,517-521,541,559,563,565,583,593,594,597,603-609,661-663,668,676,682,686,714,715

 

BCEFP 2015 – Studying for the exam

In a previous post I listed a some of the sources Brocade listed that one should use when studying for the BCEFP exam. Here I’m going through a those I found some comments on what what they are and what I think of them.

Beta Course Material

The first of the beta material available is something called “Brocade Ethernet Fabric Administration“. This is a few pdfs/slides with notes on them. Introduction of various features and components. Not much detail in the first 10 modules and basically all the modules are awfully short, some are one slide even. Hopefully this is just because it’s a beta. Progressively they become more detailed, which is good to not overwhelm the reader I guess. Checking out the data sheet for the CEF 300 course should give you some idea what you should learn after going through the materials. There are free materials available for the Ethernet Fabric Specialist Accreditation – it’s even on the tube. The youtube video is quite long but it’s an introduction to the thought behind the Ethernet Fabrics. It’s a bit outdated already I hope as they the talk talks about immaturity a lot, less than a year old. The presenter – Chip Copper – also mentions a Fabric Essentials 201 that should be out “later on down the line” – which is not out yet. Boo Urns!

Questions I got while reading material:

  • What is a hard-drop option in an extended ACL?
  • What does “override the control packet trap entries” mean? Brocade communities to the rescue. Is for normal transit traffic and traffic to the CPU == the management interface?

BCEFP Nutshell

I usually print these out, read through a few times and note down anything I don’t get so that I can go through the course materials and user guides to completely understand it. This one is vital.

Some really useful sections:

  • VCS Data Path
  • VCS Fabric – Layer 3 Routing

Some questions I needed to clarify after reading the BCEFP nutshell guide (page numbers):

  • Are there any new hardware represented in the BCEFP 2015 compared to the BCEFP 2013?
    • 6740 – 10GbE, 10GbE/FC and 40GbE ports
    • 6740T – 48 x 1/10GbE
  • VDX6720:
    • Is the VDX 6720-60 oversubscribed?
    • Is the difference between switching and forwarding bandwidth that one is how much the backplane can handle and the other is how much the ports could do?
      • Looks like that, an older version of the 6720 Data Sheet shows this, it’s been removed in a future data sheet.
  • VCS / Logical chassis / Distributed:
    • VCS Modes:
      • Logical Chassis: Requires NOS 4.0.0. Data and config paths are distributed. All is configured from the principal node.
        • Distributed
      • Fabric Cluster Mode: Data paths are distributed. Config is done independently on each node.
        • 8770 and 6740* boot up into this mode by default.
        • Local Only
    • Standalone Mode: Only compact switches support this restricted mode – 6710-6730. Only support NOS 2.1 features. Only IP static routes and in-band management.
  • VDX 8770 and what does N+1 mean? Active passive.
    • 8770-8 is N+1 with loss of one SFM
      • So it can loose one SFM and it still has a redundant SFM? Aye, this can have up to 6 SFM.
    • 8770-4 is not N+1 if one SFM is lost
      • This can have 3 SFM
  • NOS 3 requires cold reboot of standby MM during failover & firmware upgrades. Does NOS 4 do too?
  • What is an unsigned integer? – Hop Count Field in the trill frame.
    • It cannot be negative.
  • VCS features:
    • VCS Edge Port config + LACP: With Brocade type are there more models than a CNA, VDX or Brocade 8000?
    • With NOS v2.0.0a max 8 ECMP paths per switch. Different with NOS 4?
  • From show vcs detail (shows switches in the fabric):
    • What is the Internal IP used for? Unclear, the pattern is: 127.1.0.RBRIDGE ID
    • What does the state “Testing” indicate? Unclear, perhaps when running “diag *” commands?
  • show fabric
    • “show fabric islports” is similar to switchshow shows islports only, how to see device ports
      • show interface switchport # shows all ports in L2 mode (VLAN1)
    • “show fabric all” shows a short list of switches in fabric, similar to fabricshow
  • What is  “Static MAC Pre-Provisioning on vLAG” ? (p55)
  • The fibre length of a link should have deskew value of 7 microseconds. Is this configurable?
    • Looks like it’s not. It’s not in the NOS 411 cmd reference guide anyway.
  • FCOE
    • FCF = FCoE Forwarder. A switch that does both Ethernet and FC
    • ENode = FCoE Node
    • FSB =  FIP Snooping Bridge (Can I get a Yay for nested acronyms?) A FCoE Switch that needs to be connected to an FCF (p67)
    • FCoE Profiles = (p84)
  • priority-table command is just messed up. What do the numbers mean? (p66)
    • It’s a mapping of Priority Groups to Classes of Services.
  • Are Virtual Fabrics on FCoE supported these days?
    • No. FCoE needs to be on VLANs with ID < 4096.
    • Btw, Virtual Fabrics is also a feature on Ethernet. Not only FC. Used when one needs overlapping VLAN IDs – multitenancy.
  • Is the max amount of RBridges in a fabric still 24? (p77)
    • Max 24 in Logical Chassis with VDX 6710-6730. Max 32 for 6740 and 8770.
    • It is the recommended amount. Theoretical max in NOS4 is 239. One below 1111000.
  • Is there a pattern to the MAC addresses of the Switches/RBridges/FD/XD?
  • What is a VMWare Port Group?
  • In RBAC what does it mean that one can access a command but not execute it? (p86)
    • It means one can view the settings, like a ‘show command’ works but not ‘command’ to set the setting.
    • Btw: admin/user accounts are locked, only pw can be changed
  • What are these FRUs: cid-card, compact-flash, mm, SFM? (p89)
    • MM – Management Module
    • SFM – Switch Fabric Modules
    • Compact-Flash – Supposedly where the firmware/configs are stored.
    • CID-Card – Chassis ID – each card has two EEPROM – one critical and a non-critical. The non-critical can be fixed with a “CID Recovery Tool”
  • oscmd – more details about this, how to run a command? (p96)
    • oscmd arp -a
    • oscmd scp localfile remote.server:

The below I’ll bring up in a later post:

VDX Troubleshooting Course

VDX Troubleshooting Course

BCEFP practice questions / answers

http://community.brocade.com/t5/Certification/BCEFP-2013-Exam-150-180-Practice-Questions/ta-p/4099

 

Other

Intro to VCS Fabric Technology: http://www.brocade.com/downloads/documents/white_papers/intro-vcs-fabric-technology-wp.pdf
CFP- MSA CFP2 Hardweare Specs: About the 40/100Gbps CFP2 SFP. MSA – multi-source agreement.
Code names of switches? Find the NOS firmware and look in the file “platform_names”. Quite a few bird names (nighthawk, dragon, superhawk, tomahawk ;), kestrel, falcon, blackbird).

Brocade Certified Ethernet Fabric Professional 2015 Beta Exam

Intro

http://community.brocade.com/t5/Certification/BCEFP-2015-Beta-Exam-Information-and-Study-Material/ta-p/58276

The course materials, including references to various resources such as the NOS Admin guide are available on the page above.

The Advanced Ethernet Fabric Troubleshooting (VDX-TS 300-WBT) has the pdf’s on the link above, but it’s also currently free on brocade’s saba education page.

 

Objectives for 2013 exam

Objectives for the exam (2013 version, so might be different for 2015) are:

Theory and Concepts

  • Describe the VCS implementations of TRILL
  • Describe the rate-limiting features in a VCS fabric
  • Identify basic routing concepts and how they interact with an Ethernet fabric
  • Identify VDX hardware components support.

Design

  • Describe the benefits of using TRILL
  • Describe QoS in a VCS fabric
  • Demonstrate knowledge of various types of link aggregation in a VCS fabric
  • Describe VDX hardware used in the design of a VCS fabric
  • Describe AMPP concepts

Implementation and Configuration

  • Demonstrate knowledge of sharing native FC storage with FCoE devices in a VCS fabric
  • Describe the implementation of lossless Ethernet for FCoE and iSCSI traffic
  • Describe how to integrate AMPP into a vCenter environment
  • Demonstrate knowledge how to implement Layer2/Layer3 ACLs in a VCS fabric
  • Demonstrate knowledge how to configure VRRP/VRRP-E on a VDX
  • Demonstrate knowledge how to configure a VCS fabric to connect to traditional Layer 2/Layer 3 switches
  • Demonstrate knowledge how to implement vLAGs

Management

  • Demonstrate knowledge of VDX management features

Troubleshooting

  • Demonstrate advanced troubleshooting knowledge
  • Demonstrate knowledge how to troubleshoot native FCoE and VCS to FC SAN bridging
  • Demonstrate knowledge how to troubleshoot VCS to an IP network

 

 

Page numbers for 2015 beta exam below:

Network OS Administrator’s Guide v4.1.1 53-1003225-01

Pages 11-20,26-31,54-56,75-77,94,115,119,135-144,149,150,191,201-203,210,243-247,253,261,269,311-314,321,

322,338-345,361-385,400,403,406,425-433,437,465,483,495,504,506,517-521,541,559,563,565,583,593,594,597,

603-609,661-663,668,676,682,686,714,715

 

Network OS Command Reference v4.1.1 53-1003226-01

Pages 299, 1258-1260,1266,1297,1317,1318

 

Network OS v4.1.1 Brocade VDX Release Notes

Pages 4,10,28,50

 

Network OS Software Licensing Guide v4.1 53-1003164-01

Pages 11-13

 

VDX 6740 Hardware Reference Manual 53-1002829-02

Page 1
VDX 8770-4 Hardware Reference Manual 53-1002563-03

Chapter 1, Page 1; Chapter 3, Page 32

 

VDX 8770-8 Hardware Reference Manual 53-1002564-03

Chapter 1, Page 1
VDX 6730 Hardware Reference Manual 53-1002389-06

Pages 1,2,15

 

Brocade VDX 8770 Switch Data Sheet GA-DS-1701-04

 

CFP2 Hardware Specification Draft Revision 0.3

Page 46

Brocade Certified Professional Data Center Track – Check!

After ~49 posts on this blog on the topic Brocade the first larger block is finally complete: the Brocade Certified Professional Data Center Track (BCPDC)!

What’s that? So Brocade has several (4) tracks which consist of  certifications/accrediations, some are shared between the tracks and some are only in one track.
Currently, after completing 3 out of 4 you gets the title Brocade Distinguished Architect! Woop!

It took me ~3.5 years (counting since first blog post about BCFA (certified fabric administrator)) to complete all the prerequisites for BCPDC, but naturally I didn’t do it as fast as I could. I was patient and many of the certificates I got by being signing up for Brocade’s beta tests of their certs.

Not that many certificates left to take actually before I can complete another track.
Most of the remaining ones are labeled accreditations, which are unprobro_edu4_cert_pro_data_center_rgbctored tests one does at home.

  • For Brocade Certified Professional Converged Networking (BCPCN) I have 3 accrediations left (Fabric Specialist, FCoE Specialist and Ethernet Fabric Support Specialist) and 1 certification: Ethernet Fabric Professional 2013. The certification I have signed up for the free one I mentioned in an earlier blog post.
  • For Brocade Certified Professional FICON (BCPF) there’s one accrediation (Accredited FICON Specialist) an done certification (Certified Architect for FICON 2013) remaining.
  • For Brocade Certified Professional Internetworking (BCPI) there’s 3 certifications: Certified Layer 4-7 Engineer 2010, Certified Network Professional 2012 and Certified Layer 4-7 Professional 2013.

BANAS – Brocade Certification – Studying

I’m going to focus on the below things when studying for BANAS: They are based on the current objectives listed on Brocade’s page.

 

Brocade Accredited Network Advisor Specialist Exam Topics

  • The Brocade Accredited Network Advisor Specialist exam has these objectives:

Product Features

  • Demonstrate knowledge of Brocade Network Advisor product features

Installation and Configuration

  • Describe the installation and configuration of Brocade Network Advisor

  • Perform SAN Discovery

    • What are seed switches?
  • Perform IP Discovery

    • BNA 170-WBT is a course that’s currently free by Brocade – it’s about IP Discovery in BNA!
    • Once discovered devices are stored in the Management application database. First IP of the device discovered becomes the primary address of the device.
    • Simple/Profile based discovery: single: hostname/IP. Profile: range.
    • Requirements
      • Users must have Discover Setup-IP and “All IP Products AOR” privileges
        • For rediscovery only “All IP Products AOR” is needed?
      • ICMP or telnet must be enabled on devices
      • Snmpv1+v2 or v3 read-write
      • IP range of devices must be known
      • All devices must have SNMP MIB support
    • Access by: “Discover -> IP Products”.
    • One can add default username/password. One can add several and it tries the default and then the rest..
    • It uses OIDs to select products to include/exclude.
      • Cisco/Juniper are available by default.
    • Seed address: the IP the BNA server will use to contact the switches?

Migration

  • Describe considerations when migrating to Brocade Network Advisor from other tools
    • Check out the Installation Guide for BNA.

Troubleshooting

  • Demonstrate knowledge of troubleshooting Brocade Network Advisor

Brocade Certified Ethernet Fabric Professional

Brocade Certified Ethernet Fabric Professional – BCEFP – is available for free right now! I signed up and if you pass it you’re in the drawing for a $500 amazon coupon each week/month :)

Ethernet Fabric you say? “As a Brocade Certified Ethernet Fabric Professional, you must be able to demonstrate knowledge of IP, SAN, and FCoE concepts, “

What to focus on: http://community.brocade.com/t5/Certification/BCEFP-Exam-Preparation/m-p/56467

Nutshells: http://www.brocade.com/education/certification-accreditation/certified-ethernet-fabric-professional/index.page

To register for the exam, head over to the post on reddit. Basically there’s a pdf with the voucher you use when registering for the exam on pearsonvue.

Currently the “CEF 250” is free, but it’s not the main curriculum for the course.

BANAS – Brocade Accredited Network Advisor Specialist

Finally got around to start preparing the last certificate/accreditation – BANAS – to complete the Brocade Data Center Track (ok, not last. There are plenty more!).

It looks like it’s an accreditation showing that the taker can do some basic tasks in Brocade Network Advisor (BNA). This used to be a certification, so it’s probably a bit harder than it might seem!

Please note, this post is not meant to be a replacement for the official Brocade studying recommendation, just my notes on how I’m practicing for it.

Methods:

 

Install in a VM

Not much can be tested without any switches, but installing it a few times is probably helpful. Also getting acquainted with the UI and some things can still be done in the UI like:

  • Set UI options
  • Set up a firmware repository (at least import firmwares, release notes and md5 checksums)
  • Retrieve a SupportSave

Either register on Brocade’s site and get the download that way. Or get it via HP’s public page – for example here. Click on Download.
Because I’m lazy I’m installing it in a Windows 7 x64 VM 2 cores and 4GB RAM is much faster than 2GB. For just installing it you’ll need 3-4GB disk space.
Find install.exe within na1214_hp_windows.zip

The default user/password is: Administrator/password
The user/password you set during installation is for the database.

FTP/SCP/SFTP, syslog, snmp, https. Uses a postgreSQL database.

On the http/https page there are MIBs and the BNA client.

Meta Post: Blog is now also mobile friendly!

Update! If you view this blog on an a mobile device with the right user agent (and apparently very few of you do that) there’s now very little white space wasted! It’s not perfect yet but I’m working on it making it nicer. It seems to have some

Now about the normal Hemingway Ex theme, I like it. But there’s a lot of unused white areas when on 1920×1080 (17% of visitors – also the one I use ;) 

So I tweaked the stylesheets to make some sections thinner and some wider. Don’t like it? Please let me know in the comment section below.

P.s. Developer Tools in web browser are full of awesome! Trying to find the right CSS section to edit would have been mental without it.
P.s2. Just to be clear, the Blog was not hostile towards mobile users before.

Upgrade Ubuntu 12.04 LTS to 14.04

Yesterday I upgraded Ubuntu 12.04 LTS to 14.04 devel release. This is not recommended :)

I wonder what will happen when 14.04 real is out.. *update: it is out now, so the “-d” in “update-manager -d” is no longer necessary?

How to? Just run “update-manager -d” and click on some buttons!

Update process was nice, I like apt-get over yum when it sees a conflicting file it doesn’t just overwrite or write the file as .rpmsave  but instead it displays see the difference(s) between the two files and you get to decide what to do!

One issue I have heard about is VMWare Workstation 10.0.1 on 14.04 and recompiling kernel modules. Patching instructions in here: http://dandar3.blogspot.cz/ and patch file is the one at the bottom of this post https://communities.vmware.com/message/2326986

Cyanogenmod on Xperia Active (ST17i or satsuma)

Finally got fed up with the Xperia Active (ST17i or satsuma) – couldn’t update anything because all the space was used up and it was rebooting when I wasn’t touching it.
Solution? Install Cyanogenmod!

Cyanogenmod’s wiki has a good overview of how to get it installed. Basically you reinstall the operating system – but the cyanogenmod one doesn’t have all the extra crap you don’t want.

All below done from Windows 7 x64.

1. Root it! Rooted with eroot as found on http://forum.xda-developers.com/showthread.php?t=2219781

2. Install ADB. I installed the whole SDK.. apparently not needed http://forum.xda-developers.com/showthread.php?t=1474956

3. Unlock bootloader.

Poweroff the phone and get into fastboot by holding volume up while connecting the USB cable to the computer.

There is a tool called Flashtool (.net) – but it’s not necessary if you’re comfortable with the command prompt.

Also, if you can find the fastboot drivers from elsewhere that’s probably safer than installing them from flashtool (unsigned).

Run cmd as administrator if it doesn’t work or give the -waiting for device-.

Get the key from Sony via http://unlockbootloader.sonymobile.com/ – Thanks Sony!

4. Install CM 9.1.0

Or if you’re ballsy there’s CM10 the nightly build :)

Download  CM itself http://download.cyanogenmod.org/?device=satsuma and Google Apps if you want that.

Pretty straight forward after this, just follow the guide.

Use fastboot to send boot.img
Use fastboot to reboot
Your phones gets into clockwork recovery
I ran wipeall (probably should have backed up the ROM..)
Install cm9.1 from zip
Install gapps from zip

5. Conclusion

One quite annoying thing I get is: “SIM network unlock PIN” even though I’m fairly sure my phone is not locked.
First time it booted it couldn’t sign in to the phone network at all, but on second it could. I just press dismiss.. perhaps this is fixed in CM 10 Jelly Bean – but with that the phone might also be slower.
But besides that I’m quite happy with it so far. At least I won’t have to have large Office software installed anymore.
// Update a few days later: It’s been working quite nicely, not rebooting at all :)
// Update a few weeks later: DHCP on WiFi apparently stops working after a while, major bummer. But setting a fixed IP solves it. It’s apparently also possible to fix it by either setting it then back to DHCP (didn’t work for me) or by clearing out /data/misc/dhcp/* (did work for me) by opening a terminal and sending two commands: “su -” and then “rm /data/misc/dhcp/*”

Brocade – Vyatta – Future

Spoke a bit with some people in the Brocade stand at HP Discover in Barcelona. The open source / core http://www.vyatta.org/download will be kept, but could not get any commitment to what will happen to it or if they will update it. The 5400 (VSE, 6.6) and 5600 (VR, 7.x) are however available for a free 60-day trial. With the 5600 having a new architecture and is interfacing more closely with the hardware (using Intel’s DPDK to for example dedicate processes to cores) that improves performance “quite a bit”.

// Update, I since found out about VyOS: http://vyos.net/wiki/Main_Page which is a community fork of the Vyatta Core OS

Arch Linux on a T400 – Log

Trying out Arch Linux on a Lenovo T400!

First step after getting the T400 Dual Core Centrino 2.26GHz 4GB RAM: replace the 160GB spinning rust to a 120GB Samsung 840. Very easy to take out the old disk and put the new SSD in the carrier.

https://wiki.archlinux.org/index.php/Lenovo_ThinkPad_T400 has some details.

https://wiki.archlinux.org/index.php/Installation_Guide and the Beginner’s Guide are quite helpful.

To make it easy, the partition layout I created with parted was:

mklabel: MSDOS
mkpart: pri, 0%, 300M /boot ext4 – flag bootable
mkpart: pri, 300M, 100%, /, ext4
No swap!

Install base and some important packages to disk after mkfs and mounting and setting up mirrors:

pacstrap /mnt base dialog iw wpa_supplicant grub parted sudo alsa-utils vim ttf-dejavu openssh screen

Beginners’_Guide had sufficient instructions to set up the boot loader, except this was needed or the grub-mkconfig would fail with syntax error.

echo “GRUB_DISABLE_SUBMENU=y” >> /etc/default/grub

After successful boot time to get a graphical display up and running :)

Beginner’s guide to the rescue: Beginners’_Guide#Graphical_User_Interface

Lightdm was easy to set up and worked right out of the box, which GDM did not. DWM was easy too, I think I can get used to using the ABS/makepkg stuff. Nicer than compiling it and copying the binary around.

Windows key is Mod4Key in config.h for DWM.

Packages and getting chromium-pepper-flash working:

  1. Install all the dependencies for Aura
  2. Download aura and makepkg -i install it
  3. aura -A chromium-pepper-flash

alsamixer and pressing “m” mutes/unmutes a channel :)

commando.io – manage servers

https://command.io is up for public beta now. It is a page from where you can manage your servers online in a pretty web interface.

It uses ssh – you need to allow commando.io’s server to ssh into your server. You can specify with which user, which port and then it uses an ssh key to log in.

When you sign up, first you get to choose your own subdomain, like awesome.commando.io and then add your user into there. The subdomain you get, let’s say awesome.commando.io is pointing to an IP. It is with this IP that commando is connecting to your server. So this should be firewalled.

First thing I noticed was that on a CentOS 6 the command you get to copy-paste does not account for that CentOS6 sets permissions to 775 by default when running mkdir ~/.ssh.
I sent a message to commando.io with the built-in messaging tool at 13th of November at 1223. 12 minutes later I had a reply, so quite quick at responding.

Also sent in a suggestion that they look into ssh-copy-id instead of making ~/.ssh and setting permissions manually :)

Real easy to add a recipe and then run it on a server.

I could not find any existing recipes, nor where there any links to a repository or community page where one could share recipes or even example recipes.

All in all:

  • It looks nice and mostly works.
  • Is it safe? Do I want to give access to a third party provider that doesn’t have any obvious information declaring their high intent of security.
  • Some things that would be nice:
    • scheduled executions
    • using states – puppet-like, to insure that something that was done in a recipe once is still the current state of the machine.
    • group import of servers

BCvRP – Brocade Certified virtual Router Professional – Objectives

For training these I set up networks. Many.
Drawing the networks first in LibreOffice Draw and then setting them up with virtual machine templates and LAN segments.

The exam I took in October and because it was a beta exam the results aren’t out until December :)

The BCvRP has the below objectives (included for free are some of my comments on each topic).
None of this should be taken as a replacement for taking the actual course and actually doing these things on a vrouter.
And honestly, the various concepts and technologies described in the objectives below can become very complex. So before taking this course/exam you at a minimum want to know the basics of BGP and setting up an OSPF network should be a breeze.

 

OSPF Multi-Area Concepts

  • Describe OSPF routing concepts
  • Stub area – replace external routes with a default route
  • NSSA – not so stubby – can have a local external route inside a stub area
  • no-summary : exclude inter-area routes
  • LSA – link state advertisements
    • 1 All OSPFs: Lists subnets/links directly connected, does not cross area boundaries
    • 2 from DR: Lists routers connected to a network, does not cross
    • 3 from ABR: Lists networks from outside the local area
    • 4 from ASBR: Summary, lists location of ASBR
    • 5 from ASBR: AS external, list networks outside OSPF AS. 7 for NSSA.
  • Summarization: Good to have continuous addresses in an area, easier to summarize.
    • Do not summarize routes originating in Area 0.

BGP, EBGP and IBGP Concepts

  • Describe gateway protocol concepts
  • BGP Basics
    • Purpose is to determine best path (not necessarily the shortest)
    • TCP Connection, no periodic updates.
    • iBGP – within an AS / eBGP – between AS
    • Attributes – BGP policies – costs
    • eBGP – best to be on the same network
    • TCP port 179
    • A unique AS number is needed, there are private AS numbers.

eBGP

set protocols bgp AS# router-id IP
set protocols bgp AS# neighbor ip-address remote-as as-number
set protocols bgp AS# network address/mask

exact match must be in the router’s table: create a static route to blackhole on the router

iBGP = same AS on the BGP peer (the neighbor)

iBGP – a full mesh is necessary. iBGP does not forward routes learned from other iBGP peers.
One can use “next-hop-self” so that iBGP router’s change the next-hop address to a network whenever it propagates the route.
update-source – this needs to be the same as the router-id.

iBGP required settings: local AS number, neighbor address and “update source”.

bgp does not reset advertised routes after an administrator’s changes.
Changes to eBGP does not come into affect until you run the reset:
reset ip bgp external out‘. The BGP table can be large – gigabytes.
Use the word soft to only request updates and not reset the peer connection.

reset ip bgp external [ipv4 address]

 

Tuning attributes and priority

  1. Local preference – only included within an AS. Default is 100. Higher is better.
  2. AS Path – always forwarded – shorter is better
  3. Origin – lowest
  4. Multi-exit discriminator # modified by an ISP to indicate preference
  5. eBGP preferred over iBGP
  6. Lowest Peer ID
  7. Community # group of prefixes with a common property. Can be used in filters.

 

Prepending: insert your AS number in the AS in the beginning of the AS path.
Communities are created with: set policy community list

BGP troubleshooting

An active peer – not good. Trying to actively set up a session.

 

iBGP design

  • Does not have to be physically connected (as in BGP).
    • Connectivity over BGP
  • Peer to loopback address
  • Full mesh is required
    • Doesn’t scale. You can use a Route reflector (“concentrator”) and have other iBGP routers as clients.
    • route reflectors must be meshed
    • You can also create multiple private AS within your AS. Reduces members in the mesh. Called a confederation.
      • Public AS number is only visible in the config
      • The Private numbers are visible in the show ip bgp commands.

 

Create a peer group, set BGP settings on the peer group. Then assign peers to the group.

Route Redistribution

  • Describe route redistribution design and configuration
  • Best practices:
    • Set metrics
    • Do not redistribute into or out of BGP
    • Use network statements
    • Statements to direct towards BGP exit points
    • Only redistribute a network from one host (VRRP)
  • OSPF: metric type (increase cost)
  • Only active routes are redistributed

IPsec VPNs

  • Identify IKE Phase 1 and Phase 2 operations
  • Describe how to configure and troubleshoot an IPsec VPN

OpenVPN Concepts

  • Identify the features of OpenVPN
  • Describe OpenVPN configuration

VRRP Concepts

  • Describe VRRP concepts and operations

Optimization

  • Describe the attributes of WAN load balancing
  • Describe QoS features and configuration

Policy-Based Routing

  • Explain where policy-based routing falls in Brocade Vyatta packet flow
  • Configure and verify policy-based routing
  • Default: drop route entry . By default it only takes the first action that matches.
  • Rule -> Filter -> Route Map (excluding deny filters) > Take action as defined
  • Filter list: prefix 172.16.0.0/16, le 24. Any netmasks between /16 and 24, including /16.
  • Regexp for matching AS lists – use underscore to match whitespaces
  • Filter has the rules.
    • permit/deny in the filters affects if the rule is applied to the filter.
  • Route-maps has the rules.

Multicast Routing

  • Describe multicast protocols/elements
  • Configure and troubleshoot multicast routing

BCvRE – Brocade Certified virtual Router Engineer – Objectives

This post will be continuously updated with my short notes under each concept.
It’s not meant to be a replacement of the official training materials.
I’m just starting out playing with the vRouter Core / open source version and installing it in a VM and set up some networks and firewalls is probably one of the best way to learn this.
Learn by doing!

The Brocade Certified vRouter Engineer 2013 exam has these objectives:

 

Brocade Vyatta vRouter System Operations

  • Describe show command system usage
    • show – in operational mode shows status of components
    • show – in configurational mode shows the configurations
    • run show –  in configurational mode shows status of components
  • Identify key CLI operations
    • set/delete
    • copy (configs)
    • renew (new dhcp IP)
    • install (to disk)
  • Describe the commit and save processes

Ethernet Concepts

  • Identify Ethernet operations
  • Identify VLAN operations and settings
    • set interface ethernet eth0 vif <vlanid> # this creates eth0.<vlanid> a subinterface. This looks like a normal ethernet interface.
    • set interface pseudo-ethernet # these can be used if you want to set the MAC-address. Some features are not allowed for these peth devices though (VLAN, bonding).
  • Identify bonded interface operations
    • Two NICs on the same network
    • set interface bonding (IP address, mode)
    • set interface ethernet (bond-group)
  • Demonstrate knowledge of configuration and operation using show commands

TCP/IP

  • Demonstrate knowledge of the relationship between Layer 2, IP and TCP/IP
  • Identify TCD and UDP differences
  • Identify address subnets

DHCP and DNS Troubleshooting

http://www.guldmyr.com/blog/?p=2022 I’m going through how to set it up.

  • Describe troubleshooting of DHCP operations
    • show dhcp server leases
    • show log dhcp
  • Describe troubleshooting of DNS forwarding
    • monitor dns forwarding # I could not get anything into the log)
    • show dns forwarding # shows cache size for example)

Routing

http://www.guldmyr.com/blog/?p=2022 went through how to set up static routes

  • Identify uses for routing
  • Identify show commands for use with routing
  • Identify configuration of different types of static routes

Firewalls

  • Describe firewall operations and troubleshooting using show commands
  • Describe firewall rulebase operations
    • set firewall name <name> default-action
    • set firewall name <name> rule 1 destination/source
    • set firewall name <name> rule 1 action <action>
    • set interface bonding bond0 firewall in/local/out name <name>
      • in – into the router (matching on destination IP)
      • out – out from the router  (matching on source IP)
      • local – to the router itself

NAT

  • Describe NAT concepts

Upgrades

  • Describe the Brocade Vyatta upgrade process
    • 1. Install 6.5R1 to disk.
    • 2. add system image URL
    • 3. reboot
    • It is also possible to copy the config elsewhere and reinstall

Logging and Packet Captures

  • Identify logging options for firewall and NAT operations
    • set firewall name <name> rule <num> log enable
    • commit; exit
    • monitor firewall .. # and see matches to the rule.
  • Identify methods to verify operations and troubleshooting

OSPF Single-Area

http://www.guldmyr.com/blog/?p=2022 set up an area 0 OSPF

  • Describe OSPF show command output
  • Describe how to configure OSPF

BCvRE – Brocade Certified virtual Router Engineer

Been checking out the Vyatta vRouter a bit closer. Mostly because of the BCvRE exam but I’m slowly starting to think there might be some benefits to using it elsewhere too.

  1. See vyatta-a-routervpnfirewall-in-a-vm-brocade-certified-vrouter-engineer/ for where to find manuals or training materials.
  2. See the objectives.

I tried installing Vyatta vRouter 6.6 amd64 Live ISO to disk first in a Virtualbox VDI file and then uploading said file to openstack. This works, but:

Ethernet interfaces might get renamed but a startup, log in and save, poweroff and another boot should get the first interface back to eth0.

In the openstack available to me I could set up my own networking topology like this:

  • Create one network (VLAN) and define several subnets inside (these are still kind of firewalled based on IP and MACs).
  • Then create machines and add the network.
  • Power off and start the machines again (or the links stay DOWN).

VMs should see an individual eth interface per subnet.
The machines still get an IP assigned to each interface/subnet even if DHCP is disabled. If DHCP is disabled you still have to statically assign only this assigned address on the interface.
The interfaces are in order: the IP listed at the top is the IP you need to put on the first interface (eth0).

Because a lot of the things you can do with a router involves creating networks and assigning IP addresses, which openstack would block for security reasons – it was much easier to do all of these in VMWare Workstation:

DHCP/DNS

  1. Install a Vyatta VM – bridged and a private network (without a DHCP).
  2. Install another OS in a VM – this will be a client – only on the private network.
  3. Put both VMs in the same network.
  4. Configure dhcp on the Vyatta VM:
configure
delete interfaces ethernet eth1 address dhcp 
set interfaces ethernet eth1 address 10.1.1.1/24
commit

Configure dhcpd on the Vyatta VM:

configure
set service dhcp-server
set service dhcp-server shared-network-name ETH1_POOL subnet ??? # pool, dns, router

Then, set up so that the Vyatta VM routes traffic from the private network to the Internets. A NAT. This is called a source NAT in the vyatta CLI.

set nat source rule 10 ??? # Put in the settings you need. Source, outbound interface and the IP they should be seen as from the outside.

Real easy to set up a DNS forwarding server too:

set service dns forwarding listen-on eth1 
set service dns forwarding name-server 8.8.8.8
commit

Now we have a client behind the Vyatta gateway that can access the Internet!

It’s possible to set up different kinds of VPNs. For example site-to-site or remote access.

It is possible to ssh from the vyatta VM – you can even run ssh-keygen. How to add an authorized key you wonder?:

set system login user vyatta authentication ...

Routing

Another thing to test: launch a bunch of Vyatta VM and use them to route IP traffic, woop woop! The BCvRE objectives actually mention OSPF so this would be wise to test.

Starting with static routing

Key: Network Name (IP subnet, interface on the host)

  • VM hostname – Interface inside the VM: IP address

Topology:

Public (192.168.1.0/24, bridged):

  • Vyatta – eth0: 192.168.0.23

Network A (10.1.1.0/24, vmnet2):

  • Vyatta – eth1: 10.1.1.1
  • V1 – eth0: 10.1.1.10
  • V2 – eth1: 10.1.1.20

Nework B (10.2.2.0/24 , vmnet3):

  • V2 – eth2: 10.2.2.20
  • V3 – eth0: 10.2.2.30

Static routing:

Vyatta: set protocol static 10.2.2.0/24 next-hop 10.1.1.20
V1: set protocol static 10.2.2.0/24 next-hop 10.1.1.20
V3: set protocol static 10.1.1.0/24 next-hop 10.2.2.20
V3: ping 10.1.1.10

OSPF!

Adding host V4 that is in Network B and Network C.
Basically Vyatta, V2 and V4 are routers.
V1 and V3 do not run OSPF, they have their default gateway to one of their local routers.
So V3 has 10.2.2.20 and V1 has 10.1.1.1.

Public (192.168.1.0/24, bridged):

  • Vyatta – eth0: 192.168.0.23

Network A (10.1.1.0/24, vmnet2):

  • Vyatta – eth1: 10.1.1.1
  • V1 – eth0: 10.1.1.10
  • V2 – eth1: 10.1.1.20

Network B: (10.2.2.0/24, vmnet3)

  • V2 – eth2: 10.2.2.20
  • V3 – eth0: 10.2.2.30
  • V4 – eth0: 10.2.2.40

Network C: (10.3.3.0/24, vmnet4)

  • V4 – eth1: 10.3.3.40

Remove all static routes we did previously on Vyatta and V[1-2,4]:

delete protocols static route
commit
save
show proto

Set up OSPF – define the networks on each router that that router share with another router:

ALL: set loopback interface IP to something unique and with a /32
ALL: set protocols ospf redistribute connected
V4: set protocols ospf area 0 10.2.2.0/24
V2: set protocols ospf area 0 10.2.2.0/24
V2: set protocols ospf area 0 10.1.1.0/24
Vyatta: set protocols ospf area 0 10.1.1.0/24
V3: set system gateway 10.2.2.20
V1: set system gateway 10.1.1.1

Test:

V4: ping 192.168.0.23
V4: show ip ospf route

Debug:

V2: monitor protocol ospf enable lsa
V4: reboot # and wait
V2: show log|less

Vyatta: a router/vpn/firewall in a VM

Brocade has a beta exam up for BCVRE – Certified vRouter Engineer – which is on the Vyatta software from the company with the same name that Brocade bought last year.

There is the free open source core. Download from here: http://vyatta.org/downloads (no you don’t have to register).  The evaluation/subscriber version has the API and web gui available, I’ll probably check those out closer to the exam date.

I grabbed VC6.6 – Virtualization ISO. Use it in a VM and assign 5GB disk (install only requires 1G, or you could just run it on the iso, but then it doesn’t keep state between reboots) and 1GB RAM. Two NICs: One NAT and one private. But to get more acquainted with it you’ll likely have to do a bit more configuration on the hypervisor side. Such as turn off dhcpd in your virtual networks.

To install it to disk: hit “install system” at the CLI after it’s booted.

More documentation: http://docs.vyatta.com/current/wwhelp/wwhimpl/js/html/wwhelp.htm – there are descriptions how to get for example ssh management working ( set service ssh ).

The server is basically Debian with a more recent kernel (6.6 has 3.3) and a shell to make it more switch-like. It actually uses the bash completion to make it look like this. Check out /etc/bash_completion.d/vyatta-*

To remove a setting use “delete” (comparable to no in other CLIs). There is a web interface, but this is only for subscribers. Core version allows SNMP though if you want to use that :)

What to do with vyatta? A bunch of tutorials are here: http://www.vyatta.org/documentation/tips-tricks

  • NAT
  • VPN (for example connect private cloud <-> Amazon VPN)
  • Firewall
  • Routing (OSPF, BGP, etc)

But no SDN stuff (separate data and the control plane). It looks like it’s not possible to modify the flow table of a switch via Vyatta. This looks like a software router/VPN/firewall with some extras added to it.

Red Hat – Clustering and Storage Management – Course Objectives – part 2

Post 1 – http://www.guldmyr.com/blog/red-hat-clustering-and-storage-management-course-objectives/ Where I checked out udev, multipathing, iscsi, LVM and xfs.

This post is about getting using luci/ricci to get a Red Hat cluster working, but not on a RHEL machine because sadly I do not have one available for practice purposes. So CentOS64 it is. Using openstack for virtualization.

Topology: Four hosts on all three networks, -a, -b and internal. Three cluster nodes and one management node.

Get the basic cluster going:

  • image four identical nodes
  • ssh-key is distributed
  • /etc/hosts file has all hosts, IPs and networks
    • network interfaces are configured –
    • set a gateway in /etc/sysconfig/network
  • firewall
    • all traffic allowed from -a and -b networks
    • at a minimum allow traffic from the network that the hostname corresponds to that you enter in luci
  • dns (PEERDNS=no is good with several dhcp interfaces)
  • timesync with ntpd
  • luci installed on mgmt-node # ricci is a web gui
  • ricci installed on all cluster nodes # this is the service talks with corosync
    • password set for user ricci on cluster nodes
  • create cluster in luci
    • multicast perhaps doesn’t work so well in openstack ?
    • on cluster nodes this runs “yum -y install cman rgmanager lvm2-cluster sg3_utils gfs2-utils” if shared storage is selected, probably less if not.
  • fencing is really important, how to do it in openstack would require a bit of work though. Not as easy as with kvm/xvm to send a destroy domain message.

Tests:

  • Update and distribute cluster.conf
  • Have a service run on a node on the cluster (doesn’t have to have a shared storage for this).
  • Commands:
    • clustat
    • cman_tool
    • rg_test test /etc/cluster/cluster.conf start service name-of-service
    • ccs_config_validate

 

Share an iSCSI target between all nodes:

  • Using management node to share the iSCSI LUN.
  • tgtd, multipath
  • clvmd running on all nodes
  • lvmconf – make sure locking is set correctly
  • create vg with clustering
  • partprobe; multipath -r # do this often
  • vgs/lvs and make sure all nodes see the clusterd lv
  • minimum GFS filesystem is around 128M – you didn’t use all the vg right? =)
    • for testing/small cluster lowering the journal size is goodness
  • mount!

 

Red Hat – Clustering and Storage Management – Course Objectives

Attending “Red Hat Enterprise Clustering and Storage Management” in August. Quite a few of these technologies I haven’t touched upon before so probably best to go through them before the course.

Initially I wonder how many of these are Red Hat specific, or how many of these I can accomplish by using the free clones such as CentOS or Scientific Linux. We’ll see :) At least a lot of Red Hat’s guides will include their Storage Server.

I used the course content summary as a template for this post, my notes are made within them.. below.

For future questions and trolls: this is not a how-to for lazy people who just want to copy and paste. There are plenty of other sites for that. This is just the basics and it might have some pointers so that I know which are the basic steps and names/commands for each task. That way I hope it’s possible to figure out how to use the commands and such by RTFM.

 

 

Course content summary :

Clusters and storage

Get an overview of storage and cluster technologies.

ISCSI configuration

Set up and manage iSCSI.

Step 1: Setup a server that can present iSCSI LUNs. A target.

  1. CentOS 6.4 – minimal. Set up basic stuff like networking, user account, yum update, ntp/time sync then make a clone of the VM.
  2. Install some useful software like: yum install ntp parted man
  3. Add a new disk to the VM

Step 2: Make nodes for the cluster.

  1. yum install iscsi-initiator-utils

Step 3: Setup an iSCSI target on the iSCSI server.

http://www.server-world.info/en/note?os=CentOS_6&p=iscsi

  1. yum install scsi-target-utils
  2. allow port 3260
  3. edit /etc/tgt/target.conf
  4. if you do comment out the ip range and authentication it’s free-for-all

http://www.server-world.info/en/note?os=CentOS_6&p=iscsi&f=2

Step 4: Login to the target from at least two nodes by running ‘iscsiadm’ commands.

Next step would be to put an appropriate file system on the LUN.

UDEV

Learn basic manipulation and creation of udev rules.

http://www.reactivated.net/writing_udev_rules.html is an old link but just change the commands to “udevadm” instead of “udev*” and at least the sections I read worked the same.

udevadm info -a -n /dev/sdb

Above command helps you find properties which you can build rules from. Only use properties from one parent.

I have a USB key that I can pass through to my VM in VirtualBox, without any modifications it pops up as /dev/sdc.

By looking in the output of the above command I can create /etc/udev/rules.d/10-usb.rules that contains:

SUBSYSTEMS=="usb", ATTRS{serial}=="001CC0EC3450BB40E71401C9", NAME="my_usb_disk"

After “removing” the USB disk from the VM and adding it again the disk (and also all partitions!) will be called /dev/my_usb_disk. This is bad.

By using SYMLINK+=”my_usb_disk” instead of NAME=”my_usb_disk” all the /dev/sdc devices are kept and /dev/my_usb_disk points to /dev/sdc5. And on next boot it pointed to sdc6 (and before that sg3 and sdc7..). This is also bad.

To make one specific partition with a specific size be symlinked to /dev/my_usb_disk I could set this rule:

SUBSYSTEM=="block", ATTR{partition}=="5", ATTR{size}=="1933312", SYMLINK+="my_usb_disk"

You could do:

KERNEL=="sd*" SUBSYSTEM=="block", ATTR{partition}=="5", ATTR{size}=="1933312", SYMLINK+="my_usb_disk%n"

Which will create /dev/my_usb_disk5 !

This would perhaps be acceptable, but if you ever want to re-partition the disk then you’d have to change the udev rules accordingly.

If you want to create symlinks for each partition (based on it being a usb, a disk and have the USB with specified serial number):

SUBSYSTEMS=="usb", KERNEL=="sd*", ATTRS{serial}=="001CC0EC3450BB40E71401C9", SYMLINK+="my_usb_disk%n"

These things can be useful if you have several USB disks but you always want the disk to be called /dev/my_usb_disk and not sometimes /dev/sdb and sometimes /dev/sdc.

For testing one can use “udevadm test /sys/class/block/sdc”

Multipathing

Combine multiple paths to SAN devices into one fault-tolerant virtual device.

Ah, this one I’ve been in touch with before with fibrechannel, it also works with iSCSI.
Multipath is the command and be wary of devices/multipaths vs default settings.
Multipathd can be used in case there are actually multiple paths to a LUN (the target is perhaps available on two IP addresses/networks) but it can also be used to set a user_friendly name to a disk, based on its wwid.

Some good commands:

service multipathd status
yum provides */multipath.conf # device-mapper-multipath is the package. 
multipath -ll

Copy in default multipath.conf to /etc; reload and hit multipath -ll to see what it does.
After that the Fun begins!

 

Red Hat high-availability overview

Learn the architecture and component technologies in the Red Hat® High Availability Add-On.

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/High_Availability_Add-On_Overview/index.html

Quorum

Understand quorum and quorum calculations.

Fencing

Understand Fencing and fencing configuration.

Resources and resource groups

Understand rgmanager and the configuration of resources and resource groups.

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/High_Availability_Add-On_Overview/ch.gfscs.cluster-overview-rgmanager.html

Advanced resource management

Understand resource dependencies and complex resources.

Two-node cluster issues

Understand the use and limitations of 2-node clusters.

http://en.wikipedia.org/wiki/Split-brain_(computing)

LVM management

Review LVM commands and Clustered LVM (clvm).

Create Normal LVM and make a snapshot:

Tutonics has a good “ubuntu” guide for LVMs, but at least the snapshot part works the same.

  1. yum install lvm2
  2. parted /dev/vda # create two primary large physical partitions. With a CentOS64 VM in openstack I had to reboot after this step.
  3. pvcreate /dev/vda3 pvcreate /dev/vda4
  4. vgcreate VG1 /dev/vda3 /dev/vda4
  5. lvcreate -L 1G VG1 # create a smaller logical volume (to give room for snapshot volume)
  6. mkfs.ext4 /dev/VG1/
  7. mount /dev/VG1/lvol0 /mnt
  8. date >> /mnt/datehere
  9. lvcreate -L 1G -s -n snap_lvol0 /dev/VG1/lvol0
  10. date >> /mnt/datehere
  11. mkdir /snapmount
  12. mount /dev/VG1/snap_lvol0 /snapmount # mount the snapshot :)
  13. diff /snapmount/datehere /mnt/datehere

Revert a Logival Volume to the state of the snapshot:

  1. umount /mnt /snapmount
  2. lvconvert –merge /dev/VG1/snap_lvol0 # this also removes the snapshot under /dev/VG1/
  3. mount /mnt
  4. cat /mnt/datehere

XFS

Explore the Features of the XFS® file system and tools required for creating, maintaining, and troubleshooting.

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/xfsmain.html

yum provides */mkfs.xfs

yum install quota

XFS Quotas:

mount with uquota for user quotas, mount with uqnoenforce for soft quotas.
use xfs_quota -x to set quotas
help limit

To illustrate the quotas: set a limit for user “user”:

xfs -x -c "limit bsoft=100m bhard=110m user"

Then create two 50M files. While writing the 3rd file the cp command will halt when it is at the hard limit:

[user@rhce3 home]$ cp 50M 50M_2
cp: writing `50M_2': Disk quota exceeded
[user@rhce3 home]$ ls -l
total 112636
-rw-rw-r-- 1 user user 52428800 Aug 15 09:29 50M
-rw-rw-r-- 1 user user 52428800 Aug 15 09:29 50M_1
-rw-rw-r-- 1 user user 10477568 Aug 15 09:29 50M_2

Red Hat Storage

Work with Gluster to create and maintain a scale-out storage solution.

http://chauhan-rhce.blogspot.fi/2013/04/gluster-file-system-configuration-steps.html

Updates to the Red Hat Enterprise Clustering and Storage Management course

Comprehensive review

Set up high-availability services and storage.

SDN Course – Interview with Google Network Lead

This week in the SDN course on coursera there were lots of examples of real use of SDN stuff, for example like the B4 WAN by Google. They got a really interesting and cool interview with the Network Lead at Google – Amin Vahdat.
And! They actually put this interview up on youtube so you don’t have to be registered for the course on coursera to view the interview. Actually I just noticed all the interviews are there, including the one I mentioned before with the Internetz Architect David Clark.

Programming assignment for this week is to work with pyresonance, which is based on resonance + pyretic which is a controller that can change how network is forwarded/routed based on outside things, like network intrusion or bandwidth caps. This is really new stuff. The code that was put on github was put there just 3 days ago :)

Assignment is to create a load balancer and forward traffic to hosts depending on load :)