Certified OpenStack administrator – check!

Leave a reply

Yay! Took the exam last week after having studied a few days. Nothing seemed to be impossible from the list of requirements at least :)

Thought because it’s done online it can be scheduled almost on demand, but one had to wait at least 24h for the exam environment to get provisioned.

The online proctor part was a first for me. For sure it’ll help if you have a non cheaply webcam (with a longer wire) that can be moved around.

The results arrived after only a day, 96% so I missed something small somewhere. Maybe about swift if I have to guess :)

I always liked these practical exams. One really need some experience with what is being tested. I don’t it’s not possible to just study.

Playing with devstack while studying for OpenStack Certified Administrator

Leave a reply

Below I’ll go through some topics I thought about while reading through the requirements for COA:

Users and passwords because we use a LDAP at $dayjob. How to set passwords and stuff?
- openstack user password set
- openstack role add –user foo member –project demo
Users and quota. Can one set openstack to have user quota?
- guess not :)
How to default quota with CLI?
- nova quota-class commands. Found in operator’s guide in the docs.
Create openrc without horizon
- TIL that OS_AUTH in devstack is http://IP/identity . No separate port :) And couldn’t really find a nice way. After it’s working there’s an $ openstack configuration show though which tells stuff..
Cinder backup
- cool, but this service is not there by default in devstack.
Cinder encryption
- another volume type with encryption. Shouldn’t need barbican with a fixed_key but I don’t know, cinder in my devstack wasn’t really working so couldn’t attach and try it out. Have some volumes with a encryption_key_id of “000000…” so maybe? Attaching my LVMs isn’t working for some reason. Complaining about initiator ?
Cinder groups.
- Details found under cinder admin guide under rocky.. not Pike. Using cinder command one can create volume group types and then volume groups and then volumes in the volume group. All with cinder command. After you have added volumes into a group you can take snapshots of a volume group. And also create a volume group (and volumes) from the list of snapshots.
Cinder storage pool
- backends. In devstack it’s devstack@lvmdriver-1apparently one can set volume_backend_name both as a cinder.conf and as a property

Object Expiration. Supported in CEPH rados gateway? Yes, but in luminous
- available in default devstack, done with a magical header X-Delete-After:epoch
Make a Heat template from scratch using the docs.
- can be made quite minimal
Update a stack
Checking status of all the services
Forget about ctrl+w.

Study Environment

A devstack setup in an Ubuntu 18.04 in a VM in $dayjob cloud. This means no nested virtualization and I wonder how unhappy neutron will be because port security. But it’s all within one VM – it started OK, not everything worked but that’s fine with me :) Probably just need a local.conf which is not the default!

One thing I got to figure out was the LVM setup for cinder. Always fun to read logs :)

wtangy.se – now with user preferences!

Leave a reply

As part of my learning some more about modern web developments I’ve learnt that cookies now suck and one should use some kind of local storage in the web browser. One of them is Web Storage .

https://wtangy.se/ got some more updates over the weekend :)

Now if you choose a team in the /menu and later (from the same browser) visits https://wtangy.se/ you’ll get the results for that team. The selection can be cleared in bottom of the menu.

wtangy.se/menu – now with a menu!

Leave a reply

https://wtangy.se/menu has been born!

A colorful menu where one can (at least I hope) fairly easily choose a team if one does not want to change the URL :)

This was a requirement for being able to make a simple webview app :)

wtangy.se – site rename and automatic deployments!

Leave a reply

This is a good one!

Previous entries in this series: http://www.guldmyr.com/blog/wasthereannhlgamelastnight-com-now-using-object-storage/ and http://www.guldmyr.com/blog/wasthereannhlgamelastnight-appspot-com-fixed-working-again/

Renamed to wtangy.se

First things first! The website has been renamed to wtangy.se! Nobody in their right mind would type out ~~wasthereannhlgamelastnight.com~~.. so now it’s an acronym of wasthereannhlgameyesterday. wtangy.se . Using Sweden .se top level domain because there was an offer making it really cheap :)

Automatic testing and deployment

Second important update is that now we do some automatic testing and deployment.

This is done with travis-ci.org where one can view builds, the configuration is done in this file.

In google cloud there’s different versions of the apps deployed. If we don’t promote a version it will not be accessible from wtangy.se (or wasthereannhlgamelastnight.appspot.com) but via some other URL.

Right now the testing happens like this on every commit:

deploy the code to a testing version (which we don’t promote)
then we run some scripts:
1. pylint on the python scripts
2. an end to end test which tries to visit the website.
if the above succeeds we do deploy to master (which we do promote)

wasthereannhlgamelastnight.com – now using object storage!

1 Reply

To continue this series of blog posts about the awesome https://wasthereannhlgamelastnight.appspot.com/WINGS web site where you can see if there was in fact, an NHL game last night :)

Some background: First I had a python script that scraped the website of nhl.com and later changed that to just grab the data from the JSON REST API of nhl.com – much nicer. But it was still outputing the result to stdout as a set and a dictionary. And then I would in the application import this file to get the schedule. This was quite hacky and ugly :) But hey it worked.

As of this commit it now uses Google’s Cloud Object Storage:

a special URL (one has to be an admin to be able to access it)
there’s a cronjob which calls this URL once a day (22:00 in some time zone)
when this URL is called a python script runs which:
- checks what year it is and composes the URL to the API so that we only grab this season’s games (to be a bit nicer to the API)
- does some sanity checking – that the fetched data is not empty
- extracts the dates and teams as before and writes two variables,
  - one list which has the dates when there’s a game
  - one dictionary which has the dates and all the games on each date
    - probably the last would be enough ;)
- finally always overwrites the schedule

To only update it when there are changes would be cool as then I could notify myself (and possibly others) when there have been changes, but it would mean that the JSON dict has to be ordered, which they aren’t by default so I’d have to change some stuff. The GCSFileStat has a checksum-like metadata of the files called ETAG. But probably it would be best to first compute a checksum of the generated JSON and then add that as an extra metadata to the object as this ETAG is probably implemented differently between providers.

wasthereannhlgamelastnight.appspot.com – fixed – working again!

wasthereannhlgamelastnight.appspot.com – fixed – working again!

2 Replies

With NHL 2017-2018 season coming up and I had some extra spare time I thought why not finally fix this great website again :)

As NHL changed the layout of their schedule page about two seasons ago – there’s these days “infinite scrolling” or whatever it’s called when the page only loads what you see on the screen. This means it’s a bit difficult to scrape the page (but not impossible).

Lately I’ve been using REST API and JSON data for quite many things – after a short search I managed to find this hidden gem: https://statsapi.web.nhl.com/api/v1/schedule?startDate=2016-01-31&endDate=2016-02-05&expand=schedule.teams,schedule.linescore,schedule.broadcasts,schedule.ticket,schedule.game.content.media.epg&leaderCategories=&site=en_nhl&teamId=

Now that’s a link to an API provided by NHL where you get the schedule and you can filter it. I’m not sure what all the parameters do, they’re not all needed. You just need the startDate and endDate. The API also has standings and results. I have not managed to find any documentation for it. Best so far seems to be this blog post. So I’m not sure about if it’s OK to use it or if there are any restrictions.

p.s. – there is a shorter URL to the main page: https://rix.fi/nhl – but the commands – like https://wasthereannhlgamelastnight.appspot.com/MTL – does not work.

Was there an NHL game last night?

haproxy lab setup!

Leave a reply

Been seeing haproxy more and more lately as it seems even the stuff I work with are moving towards web :)

So a good time as any to play around with it!

First setup is the tag “single-node” in https://github.com/martbhell/haproxy-lab – this means it just configures one apache httpd and one haproxy. In the haproxy it creates multiple vhosts with content being served from different directories, and then it points to each of these as a haproxy backend.

To illustrate the load balancing the playbook also installs php and shows the path of the file that’s being served.

I used ansible for this and only tested it with CentOS7 in an OpenStack. The playbook also sets up some “dns” in /etc/hosts.

There are also “ops_playbooks” for disabling/enabling backends and setting weights.

I wonder what’s a good next step. Maybe multiple hosts / Docker containers? Maybe SSL termination + letsencrypt? Maybe some performance benchmarking/tuning?
I like the help for the configuration file – it begins with some detail about what an HTTP request looks like :)

Automated testing of ansible roles

Leave a reply

Contents

1 What is this?
2 How it should be done
3 How I currently to do it
4 Actual commands to test
- 4.1 Order Matters
5 How to actually set this up
6 Working Fighting with Travis

What is this?

Basic idea: whenever most things happen in your ansible repository (for example commit, pull request or release) then you want to automatically test the ansible code.

The basic tools:

syntax-checking
lint / codying style adherence
actually running the code
is it idempotent
does the end result look like you want it to?

How it should be done

Use something like molecule https://github.com/metacloud/molecule which can launch your container/virtual machine, run ansible, check for lint and also run some testing framework like serverspec/testinfra.

How I currently to do it

I use travis to test many ansible roles and playbooks. From travis you basically get an Ubuntu machine and in that you can run whatever you want.

Basic process I’ve used for ansible testing:

Configure docker on the Ubuntu machine (or LXC in some roles)
Launch a docker with the OS you want to test on (in my case mostly CentOS 7, but sometimes Debian)
Run ansible-playbook with –syntax-check, –check and twice to check for idempotency
Run some manual commands at the end to test whatever was configured / or at least print some config files to make sure they look OK

All of the above and more should be doable now with molecule, first and last time I tried I couldn’t get it to work but it’s looking better.

Actual commands to test

ansible-playbook –syntax-check
ansible-lint
ansible-playbook
ansible-playbook
ansible-playbook –check

Order Matters

Do you want to run it in noop mode ( –check ) before or after the role has first run at least once to configure all the things?

How to actually set this up

Official travis documentation

Login with your github account on travis.org (or travis.com if it’s a private repo) ( and connect your github organization ).

Enable the repository, for example https://travis-ci.org/CSCfi/ansible-role-dhcp_server

Add some files to your repo. I usually copy .travis.yml and tests/ directory from an existing repository like ansible-role-cvmfs .

Modify the test playbook – tests/test.yml to include the new role, maybe change some default variables and have a look in test-in-docker-image.sh script if there are anything you want to add or remove from there too.

Push to github and watch the build log :)

Working Fighting with Travis

Fighting with docker took a lot of my time when getting this working the first time. Especially as I use ansible to configure servers that run multiple services and want to have a full systemd inside the container.

Commands to run on an Ubuntu 14.04 VM to get a kind of similar environment as in travis:

sudo apt update
sudo apt upgrade
sudo apt install build-essential libssl-dev libffi-dev python-dev git
sudo apt install docker.io cgroup-lite
/usr/share/docker.io/contrib/check-config.sh 
echo 'DOCKER_OPTS="-H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock -s devicemapper"' | sudo tee /etc/default/docker > /dev/null
sudo cgroups-mount

And then from there run the commands you have in .travis.yml

linux.conf.au.2016 and a FreeIPA workshop

Leave a reply

https://github.com/freeipa/freeipa-workshop

In preparation for the RH414 course I’m taking next week I think I should have a look at kerberos, freeipa and bind a bit :)

During linux.conf.au.2016 there was a workshop on FreeIPA. (There were many other interesting talks there, for example the Network Performance Tuning by Jamie Bainbridge).

There is a video to accompany it: https://www.youtube.com/watch?v=VLhNcirKFDs

Notes

Bonus feature: get acquainted with vagrant too!

Vagrant 1.7.4 and Virtualbox 5.0 works just fine together (except I had some issues with network interfaces on Ubuntu 15.10 and Virtualbox 5 and Vagrant – the MAC addresses were the same on the VM’s interfaces to the “NAT” network- they also got some weird IP addresses there). I could only find that IP used in resolv.conf (from the dhcp) – so that could be changed.

RH413 – Red Hat Server Hardening

Leave a reply

I’m attending this training in a week or so. This post will be updated as I go through the sections I want to check out before the training starts.

https://www.redhat.com/en/services/training/rh413-red-hat-server-hardening

Track security updates
- Understand how Red Hat Enterprise Linux produces updates and how to use yum to perform queries to identify what errata are available.
Manage software updates
- Develop a process for applying updates to systems including verifying properties of the update.
Create file systems
- Allocate an advanced file system layout and use file system encryption.
Manage file systems
- Adjust file system properties through security related options and file system attributes.
Manage special permissions
- Work with set user ID (SUID), set group ID (SGID), and sticky (SVTX) permissions and locate files with these permissions enabled.
Manage additional file access controls
- Modify default permissions applied to files and directories; work with file access control lists.
Monitor for file system changes
- Configure software to monitor the files on your machine for changes.
Manage user accounts
- Set password-aging properties for users; audit user accounts.
Manage pluggable authentication modules (PAMs)
- Apply changes to PAMs to enforce different types of rules on users.
Secure console access
- Adjust properties for various console services to enable or disable settings based on security.
Install central authentication
- Install and configure a Red Hat Identity Management server and client.
Manage central authentication
- Configure Red Hat Identity Management rules to control both user access to client systems and additional privileges granted to users on those systems.
Configure system logging
- Configure remote logging to use transport layer encryption and manage additional logs generated by remote systems.
Configure system auditing
- Enable and configure system auditing.
Control access to network services
- Manage firewall rules to limit connectivity to network services.

From the exam https://www.redhat.com/en/services/training/ex413-red-hat-certificate-expertise-server-hardening-exam

Identify Red Hat Common Vulnerabilities and Exposures (CVEs) and Red Hat Security Advisories (RHSAs) and selectively update systems based on this information
Verify package security and validity
Identify and employ standards-based practices for configuring file system security, create and use encrypted file systems, tune file system features, and use specific mount options to restrict access to file system volumes
Configure default permissions for users and use special file permissions, attributes, and access control lists (ACLs) to control access to files
Install and use intrusion detection capabilities in Red Hat Enterprise Linux to monitor critical system files
Manage user account security and user password security
Manage system login security using pluggable authentication modules (PAM)
Configure console security by disabling features that allow systems to be rebooted or powered off using bootloader passwords
Configure system-wide acceptable use notifications
Install, configure, and manage identity management services and configure identity management clients
Configure remote system logging services, configure system logging, and manage system log files using mechanisms such as log rotation and compression
Configure system auditing services and review audit reports
Use network scanning tools to identify open network service ports and configure and troubleshoot system firewalling

Let’s encrypt the web – renewal

1 Reply

So easy!

just:

As I ran the letsencrypt-auto last time, I did again.

sudo systemctl stop nginx
cd letsencrypt
git pull
./letsencrypt-auto
enter enter etc
sudo apache2ctl stop # .. why did it start apache2 automatically?
sudo systemctl start nginx

Since letsencrypt-auto version 0.5.0 it’s:

sudo systemctl stop nginx
cd letsencrypt
git pull
./letsencrypt-auto –standalone –domains “my.example.com,2.example.com”
sudo systemctl restart nginx

Since certbot-auto (renamed from letsencrypt):

sudo systemctl stop nginx
./certbot-auto renew
sudo systemctl start nginx

let’s encrypt the web!

1 Reply

Letsencrypt is finally in public beta!

Got from ssllabs.com https enabled on my own play webhost today with let’s encrypt!

There are many good guides for getting this setup. This is how I got it working with nginx (without using the experimental nginx plugin of letsencrypt).

on the webhost (not as root):

git clone https://github.com/letsencrypt/letsencrypt
letsencrypt-auto
#eventually this generates some certificates into /etc/letsencrypt
#of course you should read scripts before running anything, there are for example acme-tiny, gethttpsforfree.com and letsencrypt-nosudo that might be better.
#mozilla has some server side SSL recommendations on https://wiki.mozilla.org/Security/Server_Side_TLS

Modify your nginx site file to have something like this:

server {
 listen [::]:443 ssl ipv6only=off;

ssl on;
 ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

ssl_session_cache shared:SSL:50m;
 ssl_session_timeout 5m;
 ssl_session_tickets off;

ssl_protocols TLSv1.1 TLSv1.2;
 ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK';
 ssl_prefer_server_ciphers on;

ssl_dhparam /etc/nginx/dhparams.pem;

# ssl_stapling on;
# ssl_stapling_verify on;
# resolver 193.166.4.24; 
 root /var/www;
 index index.html index.htm index.php;

# Make site accessible from http://localhost/
 server_name localhost;

add_header Strict-Transport-Security "max-age=15724800";

Was there an NHL game last night?

1 Reply

Yesterday my Internet activities was restricted unnecessarily!

While waiting for the replay of last night’s NHL game to air, I didn’t want to browse quite a large chunk of my normal Internets – because knowing the score while watching the game sucks. Unbeknownst to me – there was no game last night! Queue impatience, etc.

No more! (at least for the remainder of 2015 edition of the Stanley Cup).

Introducing: http://wasthereannhlgamelastnight.appspot.com/

Today (2015-06-07) it says YES, hopefully tomorrow (2015-06-08) it will say NO :) //update – it did!

This is my first trek into google cloudappengine thingy. Very much work in progress but it’s enough for now.

check_irods – nagios plugin to check the functionality of an iRODS server

Leave a reply

Part of my $dayjob as a sysadmin is to monitor all things.

Today I felt like checking if the users on our servers could use the local iRODS storage and thus check_irods was born!

It checks if it can:

put
list
get
remove

a temporary file.

Dependencies:

iRODS 3.2 with OS trusted authentication
mktemp

Source: https://github.com/martbhell/nagios-checks/tree/master/plugins/check_irods

Nagios Health Check of a DDN SFA12K

Leave a reply

Part of my $dayjob as a sysadmin is to monitor all things.

I’ll be publishing my home-made nagios checks on github in the near future.

Here is the first one that uses the Web API of a DDN’s SFA12K (might work on the 10k too, haven’t tried) which is a storage platform.

The URL to the check is located here: https://github.com/martbhell/nagios-checks/tree/master/plugins/check_ddn

Unfortunately it seems that the Python Egg (the library / API bindings) is still not available online so one has to ask DDN Support to get that.

It’s not perfect, there’s much room for improvement, refactoring, moving the password/username out of a variable and it makes many assumptions.
But making it work for you shouldn’t be too hard. If you have any questions comment here or on github :)

High amount of Load_Cycle_Count on Green Western Digital disks

Leave a reply

You are monitoring the SMART values of your disks right? They’re usually a real good indicator of the health of the drive.

Thought I’d check out the SMART value of the disks in my desktop today (while checking if I had notifications from smartd on).

Low and behold, the Load_Cycle_Count (LLC) was really high, much higher than power_cycle_count on the 3TB WD disk I have. It turns out this is quite an old problem so there are a few posts about this on the Internets.
The Interwebs says max in the specs are 300k load cycles. Smartctl -a says I’m already at 218602 after 9302 power on hours (387 days but I power off the computer at night).

Disk:

Model Family:     Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model:     WDC WD30EZRX-00DC0B0

For Windows there’s a wdidle3.exe that is a DOS program that one can put on a bootable floppy (…) and boot a computer on to change some stuff on a disk.

Fortunately I run Linux (Ubuntu 14.10 since yesterday) and there’s a tool called idl3ctl – one can grab it from here: http://idle3-tools.sourceforge.net/

I got the latest source code and compiled it myself because there had been some updates to it since the last release (2012 vs 2011 ..).
“idl3ctl -g” shows that the disk was set to park itself after 8s. I disabled that with idl3ctl and powered off and on the computer and now the tool says it’s disabled.

Hopefully this should increase the lifetime of my disk.

Finnish words from ankidroid

Leave a reply

Lately I’ve been using ankidroid to study Finnish – or at least to expand my vocabulary a bit. I think it’s great but it won’t work by itself and requires quite a bit of tenacity.

The good Finnish deck:https://ankiweb.net/shared/info/1918695216 it’s called “Finnish järjestyksessä”.

It’s made by Rendall and it’s quite good, it took a list of the most common Finnish words (assembled by my company ;)

The deck won’t work by itself though, I get a lot of help by:

Get the word in a sentence helps tremendously for learning it. Just learning the words helps a lot, especially if you don’t care so much about using the exact right form. Learning how to use them is also important, and that’s where talking or reading things help.
Asking someone for assistance or clarifications about words is important. For example you might have words that are translated to the same word in English but you’d use them differently in Finnish.

Some notes about the contents:

Some translations are not awesome or the first word that shows up is the archaic meaning. So for those you’d want to click the edit card and change the meaning to something modern.
Some erisnimi words are just fairly useless unless you want to learn a city /person name, like NATO, Salo, Joensuu, Jorma or Jukka. They might have a different meaning besides the city name but they’re rarely/never used for that meaning.

Making changes to the template:

On each card there is this “info” and sometimes hints. I found myself wanting to change the URL to wiktionary on the cards because google chrome didn’t redirect wiktionary.org to en.wiktionary org but to www.m.wiktionary.org which doesn’t exist. It’s quite easy to change this, because that part is done with a template and is not added in each card. To change:

synch your decks
install anki on your compute
login with your account and synch it
browse the deck and on the right-hand side you’ll see your decks and the ones you have marked and so on, there’s also a few called “adjektiivi”, “prononimit” and so on. These are the templates.
Click on a template and then on the “Cards…” button.
This will show “Card 1” and “Card 2”. In the back template you can then change the URL to whatever you want!

BCEFP 2015 certified!

Leave a reply

Passed the Brocade Certified Ethernet Fabric Professional 2015 exam in May and I finally got the results back!

https://www.certmetrics.com/brocade/public/transcript.aspx?transcript=8XRF1FE12MR41GC4

This one felt quite hairy compared to the other tests I’ve taken. Definitely recommend doing the course / getting some real hands-on experience for these certifications.

Lustre 2.5 + CentOS6 Test in OpenStack

Leave a reply

Reason: Testing to Lustre 2.5 from a clean CentOS 6.5 install in an openstack.

Three VMs: two servers, one MDS, one OSS and one Client. CentOS65 on all. An open internal ethernet network for the lustre traffic (don’t forget firewalls). Yum updated to latest kernel. Two volumes presented to the lustreserver and lustreoss for MDT + OST, both are at /dev/vdc. Hostnames set. /etc/hosts updated with three IPs: lustreserver, lustreoss and lustreclient.

With 2.6.32-431.17.1.el6.x86_64 there’s some issues at the moment for building the server components. One needs to use the latest branch for 2.5 so the instructions are https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821

Server side

MDT/OST: Install e2fsprogs and reboot after yum update (to run the latest kernel kernel).

yum localinstall all files from: http://downloads.whamcloud.com/public/e2fsprogs/1.42.9.wc1/el6/RPMS/x86_64/

Next is to rebuild lustre kernels to work with the kernel you are running and the one you have installed for next boot: https://wiki.hpdd.intel.com/display/PUB/Rebuilding+the+Lustre-client+rpms+for+a+new+kernel

RPMS are here: http://downloads.whamcloud.com/public/lustre/latest-feature-release/el6/server/SRPMS/

For rebuilding these are also needed:

yum -y install kernel-devel* kernel-debug* rpm-build make libselinux-devel gcc

basically:

git clone -b b2_5 git://git.whamcloud.com/fs/lustre-release.git
autogen
install kernel.src from redhat (puts tar.gz in /root/rpmbuild/SOURCES/)
if rpmbuilding as user build, then copy files from /root/rpmbuild into /home/build/rpmbuild..
rebuilding kernel requires quite a bit of hard disk space, as I only had 10G for / then I made symlinks under $HOME to the $HOME/kernel and $HOME/lustre-release

yum -y install expect and install the new kernel with lustre patches and the lustre and lustre modules.

Not important?: WARNING: /lib/modules/2.6.32-431.17.1.el6.x86_64/weak-updates/kernel/fs/lustre/fsfilt_ldiskfs.ko needs unknown symbol ldiskfs_free_blocks

/sbin/new-kernel-pkg –package kernel –mkinitrd –dracut –depmod –install 2.6.32.431.17.1.el6_lustre

chkconfig lustre on

edit /etc/modprobe.d/lustre.conf and add the lnet parameters

modprobe lnet
lctl network up
# lctl list_nids

creating MDT: mkfs.lustre –mdt –mgs –index=0 –fsname=wrk /dev/vdc1
mounting MDT: mkdir /mnt/MDT; mount.lustre /dev/vdc1 /mnt/MDT

creating OST: mkfs.lustre –ost –index=0 –fsname=wrk –mgsnode=lustreserver /dev/vdc1
mounting OST: mkdir /mnt/OST1; mount -t lustre /dev/vdc1 /mnt/OST1

Client Side

rpmbuild --rebuild --without servers

cd /root/rpmbuild/RPMS/x86_64
rpm -Uvh lustre-client*

add modprobe.d/lustre.conf
modprobe lnet
lctl network up
lctl list_nids

mount.lustre lustreserver@tcp:/wrk /wrk

lfs df!

BCEFP 2015 – Studying for the exam – part 3

Leave a reply

This third post focuses on the remaining sources of information I had for studying for the BCEPF. At the time this post is published I have taken the exam.

When I make comments to CLI commands I put them after a #.

This is part of a series of posts on the topic of studying for Brocade’s Certified Ethernet Fabric Professional.

The two previous posts: Objectives and reading materials, course and nutshell guide and NOS Admin Guide

Contents

1 VDX Troubleshooting Course
2 BCEFP practice questions / answers
3 Other

VDX Troubleshooting Course

The material available also feels very short, same as the beta material available for the CEF300 , like only the parts of the slides that were updated for the BCEFP 2015 beta were included.
When a slide says “(cont.)” but there was no previous slides on this topic, that’s a hint :)
Take the (currently free) course on Brocade’s SABA – it’s under Education on my.brocade.com. It has way more slides and info.

Some notes from the course:

Firmware Upgrade

Can upgrade all/selected RBridges in a logical chassis: firmware download logical-chassis
FTP/SCP/SFTP/USB(only local switch with USB)
By default it stages firmware only – so no reboot or activate. By adding auto-activate it reboots all RBRidges at the same time, not recommended.

SNMP

When BNA discovers a switch it automagically configures the switch to send traps (UDP 162) to the BNA server.

Fabric Formation:

Requires: Licenses. Same VCS ID, unique RBridge ID and same VCS mode (Fabric Cluster or Logical Chassis)
Check:
- ISL ports are operational (show fabric islports)
- Incompatible Firmware Levels

ISLs:

no fabric isl enable # this disables ISL formation. This makes it an edge port
CPU could be too busy to send ISL keepalives
If ISL is segmented and interface is up/up – it’s probably a config issue.

vLAGs:

show running-config interface TenGigabitEthernet 1/0/2 # shows config
- no shutdown
- channel-group $NUMBER mode active type standard # active – LACP. Standard/Brocade proprietary.
show interface TenGigabitEthernet 1/0/2 # shows status
- When counters are non-zero and looking for errors. Clear them and compare the delta.

Other:

show interface stats brief # shows discards, errors and CRC
VRRP:
- show vrrp detail
- pre-empting : if a virtual router comes online with higher priority than the current it will take over
- VRRPE: Can enable short-path-forwarding. If one of the backup virtual routers (that don’t own the Virtual IP) can actually forward traffic if that is advantageous.

FCoE:

show running-config zoning # show FCoE zoning
show fabric all #
- RBRidge with this name: fcr_fd_160 # this comes online when fabrics are connected and Fibre Channel Routing is used.
- RBRidge with this name: fcr_xd_4_100 # this comes online when devices across FC Fabrics can communicate. Don’t see this? Check zoning.

iSCSI:

BCEFP practice questions / answers

http://community.brocade.com/t5/Certification/BCEFP-2013-Exam-150-180-Practice-Questions/ta-p/4099

These are decent practice questions and is nice because the answers give some explanation to the answers too.

Other

Intro to VCS Fabric Technology: http://www.brocade.com/downloads/documents/white_papers/intro-vcs-fabric-technology-wp.pdf

CFP- MSA CFP2 Hardweare Specs:

about the 40/100Gbps CFP2 SFP. MSA – multi-source agreement.
CFP2 module shall support LC, MTP12 and MTP24 optical connector types. MPO

NOS 4.1.1 release notes (p4,10,28,50):

4.1.0 and later support VRRP-E across VCS fabrics.
4.1.0 and later have vlag ignore split on by default
clear mac-address-table can clear MAC addresses associated with vLAGs and on other switches
Page 50 Has a table of scalability numbers for various features such as (6710 VCS, 6740 VCS, 8770 VCS):
- max members of a LAG (8,16,8)
- max switches in a fabric/logical cluster (24,32,32)
- max ECMP paths (8,8,16)
- max member ports in a vLAG (64)
- max member of VMs (8k)
- max ARP entries (8k,12,50k)

Network OS Command Reference v4.1.1 53-1003226-01

Pages 299, 1258-1260,1266,1297,1317,1318

firmware download
snmp-server user # access
snmp-server v3host # trap recipients
spanning-tree edgeport # quickly transitions to forwarding state: only for RSTP/MSTP. Portfast for STP.
switchport access # only allows untagged and priority tagged
switchport trunk allowed vlan ${rspan-vlan} # add allowed VLAN on trunks on L2 interfaces in trunk mode
switchport trunk default-vlan # put all non-matching traffic into this VLAN

Hardware reference manuals

VDX 6740 Hardware Reference Manual 53-1002829-02: Page 1

6470: 24 1/10GbE SFP+ ports.
6740T: 24 RJ-45
6740-1G: 48 RJ-45 Base-T. 10Gb with license.

VDX 8770-4 / 8770-8 Hardware Reference Manual 53-1002563-03:

Chapter 1, Page 1:
- Features CloudPlex.
- Requires NOS 3.0.0 or greater.
- 8770-8:
  - Up to 384 10GbE or 96 40GbE. Dual MM. 6 SFM. Max 8 PSU. 4 Fans. SX or LX 1Gbps SFP transceivers.
- 8770-4:
Chapter 3, Page 32
- For copper connections to < 1Gbps BaseT switches a crossover cable is needed (but it might not be if MDI/MDIX works..).
- LC connectors for fiber ports

VDX 6730 Hardware Reference Manual 53-1002389-06: Pages 1,2,15

6730-32: 32-ports. 6730-76: 76 ports. 8 or 16 x 8GB FC ports.

Network OS Software Licensing Guide v4.1 53-1003164-01

Pages 11-13

All have FCoE license (except 6710).
All have POD licenses (except 8770)
6740 have 10/40GbE port upgrades
8770 have L3 and Advanced Services

Notes:

for multi-hop FCoE it is needed on each node
L3: OSFP, VRRP, PIM-SM, Route-Maps, prefix list
Advanced: FCoE and L3
After installing a time-based license you cannot change system date or time. NTP is however not blocked. If you are using NTP, don’t change system date/time when a time-based license is installed.

BCEFP 2015 – Studying for the exam – part 2

Leave a reply

This second post focuses on the NOS Admin Guide.

When I make comments to CLI commands I put them after a #.

This is part of a series of posts on the topic of studying for Brocade’s Certified Ethernet Fabric Professional.

The two previous posts: Objectives and reading materials and course and nutshell guide

The NOS 4.1.1 Admin Guide

I’ve been reading the pages on paper (together with a highlighter :) that I printed with the help of my script below and there is lots of goodness in there.
For sure some topics are brought up without any preamble so for these I just make a note in the paper that I need to check out this other thing later.
Especially the Fibre Channel things take up quite a lot of pages. I thought in these devices FC would not be with so much focus but it seems like they do re-use a lot of the things in FC that works.

Notes and acronyms (page in NOS Admin Guide):

DCB – lossless. Able to allocate bandwidth on links.
TRILL – transparent interconnections of lots of links.
RBridge – Routing Bridge. Lowest WWN or priority.
Looks like on p54 only the text about Logical Chassis cluster config is applicable.
Trunking between VDX8770 and B8000 are not supported (B8000 is some early version of FCoE from Brocade, not visible on Brocade’s page where they list their switches)
ECMP – Equal-cost multi-path routing (p149)
AG – VCS must be enabled for Access Gateway
AMPP – Automatic Migration of Port Profiles – some OK pictures around p375
VRF – Virtual Routing and Forwarding

Questions:

There is also a Openstack Neutron Plugin (p29)
Would be good to include also page 114 before page 115 to see what they mean with leaf/spine/core (p115)
OOB access to console is via serial (p115)
How to reload a group of switches? (p115)
- reload system rbridge-id all
Does trill use IS-IS type link-state? (p136) Yes
Can VF_Ports be anywhere in the fabric? (p202) Yes, they must be mapped to N_Ports.
Is there no web interface on the VDX? (p269) Probably not, there are some “http server” and “ip http-server” commands.
What are valid upgrade paths? Not so clear. 3.0.0 to 4.0.0 is not OK. 3.0.1 to 4.0.0 is OK. (p341)
What is this netinstall? (p371) – 10 hits on google: brocade “netinstall” vdx
What does the asterisk mean in the output of “do show vcs” ? (p597) The one you are running the command on? Is not principal RBRidge, that is >.

Commands (# comments) (page):

backup config: copy rbridge-running-config rbridge-id rbridge-id location_config
- copy rbridge-running-config rbridge-id 2 scp://user:pw@host
vcs
- no vcs logical-chassis enable # remove a node from logical chassis cluster (p76)
- vcs replace rbridge-id 3 # replace RBridge with id 3 (p77)
- enable (p139)
- virtual ip address 10.1.1.1 (p143)
config terminal # to enter global exec mode (p94)
firmware download (p119)
logical-chassis principal-switchover (p138)
- and logical-chassis principal-priority are the only logical-chassis commands
disabling a port:
- shutdown # on an ISL brings down link and FSPF adjacency.
- no fabric isl enable # link stays up, shorter reconvergence
show
- vcs virtual-ip (p143)
vcenter/vnetwork # used to connect to a vcenter and to discover hosts. (p243)
bind # create persistent binding between logical FCoE port and 10G/40G/LAG port. Port or MAC, not both. (p345)
enable statistics direction # for VXLAN tunnels to enable statistics on VLANs. (p365)
no spanning-tree shutdown # default for all VLANs – meaning it’s enabled. (p381)
lacp system-priority 25000 # For deciding which system is in charge of resolving LAG conflicts. (p437)
nas server-ip IP/PREFIX # Set IPs for AutoQoS for NAS (p506)
address-family ipv4 unicast # Used to enter IPv4 config in a VRF (p609)
debug lacp pdu # turn on debug (p714)
- terminal monitor # view debug messages in terminal

Printing the NOS Admin Guide relevant pages:

Because the slides for the BCEFP course were insufficient I would get a lot of the basic information about the NOS from the NOS Admin Guide.
In the materials provided the NOS Admin Guide was separated into two documents. The guide is of course available in one pdf. Go to the web version and click on the pdf icon.
This makes printing based on the numbers provided easier. However the NOS Admin Guide for v4.1.1 referenced was one version below the one on the html version.

Now the numbers referenced are the numbers in the document, not the one told by the pdf viewer. So actually page 11 is page 13. Page 135 is 137. 311 is 313. 425 is 427. 517 is 519. 661 is 663. 714 is 716.
I checked a few to make sure there were no major increase due to version difference or elsewhere. One could with a bit of scripting increase each number with two like:

1,13-22,28-33,56-58,77-79,96,117,121,137-146,151,152,193,203-205,212,245-249,255,263,271,313-316,323,324,340-347,363-387,402,405,408,427-435,439,467,485,497,506,508,519-523,543,561,565,567,585,595,596,599,605-611,663-665,670,678,684,688,716,717

Cover page added to make it look nicer when printing. Old numbers:

Network OS Administrator’s Guide v4.1.1 53-1003225-01

Pages 11-20,26-31,54-56,75-77,94,115,119,135-144,149,150,191,201-203,210,243-247,253,261,269,311-314,321,322,338-345,361-385,400,403,406,425-433,437,465,483,495,504,506,517-521,541,559,563,565,583,593,594,597,603-609,661-663,668,676,682,686,714,715

BCEFP 2015 – Studying for the exam

3 Replies

In a previous post I listed a some of the sources Brocade listed that one should use when studying for the BCEFP exam. Here I’m going through a those I found some comments on what what they are and what I think of them.

Contents

1 Beta Course Material
2 BCEFP Nutshell
3 VDX Troubleshooting Course
4 BCEFP practice questions / answers
5 Other

Beta Course Material

The first of the beta material available is something called “Brocade Ethernet Fabric Administration“. This is a few pdfs/slides with notes on them. Introduction of various features and components. Not much detail in the first 10 modules and basically all the modules are awfully short, some are one slide even. Hopefully this is just because it’s a beta. Progressively they become more detailed, which is good to not overwhelm the reader I guess. Checking out the data sheet for the CEF 300 course should give you some idea what you should learn after going through the materials. There are free materials available for the Ethernet Fabric Specialist Accreditation – it’s even on the tube. The youtube video is quite long but it’s an introduction to the thought behind the Ethernet Fabrics. It’s a bit outdated already I hope as they the talk talks about immaturity a lot, less than a year old. The presenter – Chip Copper – also mentions a Fabric Essentials 201 that should be out “later on down the line” – which is not out yet. Boo Urns!

Questions I got while reading material:

What is a hard-drop option in an extended ACL?
What does “override the control packet trap entries” mean? Brocade communities to the rescue. Is for normal transit traffic and traffic to the CPU == the management interface?

BCEFP Nutshell

I usually print these out, read through a few times and note down anything I don’t get so that I can go through the course materials and user guides to completely understand it. This one is vital.

Some really useful sections:

VCS Data Path
VCS Fabric – Layer 3 Routing

Some questions I needed to clarify after reading the BCEFP nutshell guide (page numbers):

Are there any new hardware represented in the BCEFP 2015 compared to the BCEFP 2013?
- 6740 – 10GbE, 10GbE/FC and 40GbE ports
- 6740T – 48 x 1/10GbE
VDX6720:
- Is the VDX 6720-60 oversubscribed?
- Is the difference between switching and forwarding bandwidth that one is how much the backplane can handle and the other is how much the ports could do?
  - Looks like that, an older version of the 6720 Data Sheet shows this, it’s been removed in a future data sheet.
VCS / Logical chassis / Distributed:
- VCS Modes:
  - Logical Chassis: Requires NOS 4.0.0. Data and config paths are distributed. All is configured from the principal node.
    - Distributed
  - Fabric Cluster Mode: Data paths are distributed. Config is done independently on each node.
    - 8770 and 6740* boot up into this mode by default.
    - Local Only
- Standalone Mode: Only compact switches support this restricted mode – 6710-6730. Only support NOS 2.1 features. Only IP static routes and in-band management.
VDX 8770 and what does N+1 mean? Active passive.
- 8770-8 is N+1 with loss of one SFM
  - So it can loose one SFM and it still has a redundant SFM? Aye, this can have up to 6 SFM.
- 8770-4 is not N+1 if one SFM is lost
  - This can have 3 SFM
NOS 3 requires cold reboot of standby MM during failover & firmware upgrades. Does NOS 4 do too?
What is an unsigned integer? – Hop Count Field in the trill frame.
- It cannot be negative.
VCS features:
- VCS Edge Port config + LACP: With Brocade type are there more models than a CNA, VDX or Brocade 8000?
- With NOS v2.0.0a max 8 ECMP paths per switch. Different with NOS 4?
From show vcs detail (shows switches in the fabric):
- What is the Internal IP used for? Unclear, the pattern is: 127.1.0.RBRIDGE ID
- What does the state “Testing” indicate? Unclear, perhaps when running “diag *” commands?
show fabric
- “show fabric islports” is similar to switchshow shows islports only, how to see device ports
  - show interface switchport # shows all ports in L2 mode (VLAN1)
- “show fabric all” shows a short list of switches in fabric, similar to fabricshow
What is “Static MAC Pre-Provisioning on vLAG” ? (p55)
The fibre length of a link should have deskew value of 7 microseconds. Is this configurable?
- Looks like it’s not. It’s not in the NOS 411 cmd reference guide anyway.
FCOE
- FCF = FCoE Forwarder. A switch that does both Ethernet and FC
- ENode = FCoE Node
- FSB = FIP Snooping Bridge (Can I get a Yay for nested acronyms?) A FCoE Switch that needs to be connected to an FCF (p67)
- FCoE Profiles = (p84)
priority-table command is just messed up. What do the numbers mean? (p66)
- It’s a mapping of Priority Groups to Classes of Services.
Are Virtual Fabrics on FCoE supported these days?
- No. FCoE needs to be on VLANs with ID < 4096.
- Btw, Virtual Fabrics is also a feature on Ethernet. Not only FC. Used when one needs overlapping VLAN IDs – multitenancy.
Is the max amount of RBridges in a fabric still 24? (p77)
- Max 24 in Logical Chassis with VDX 6710-6730. Max 32 for 6740 and 8770.
- It is the recommended amount. Theoretical max in NOS4 is 239. One below 1111000.
Is there a pattern to the MAC addresses of the Switches/RBridges/FD/XD?
What is a VMWare Port Group?
- Found in VMWare Virtual Networking Concepts
- Important with VMotion, makes sure that when a VM moves to a different hypervisor it gets the same network configuration.
In RBAC what does it mean that one can access a command but not execute it? (p86)
- It means one can view the settings, like a ‘show command’ works but not ‘command’ to set the setting.
- Btw: admin/user accounts are locked, only pw can be changed
What are these FRUs: cid-card, compact-flash, mm, SFM? (p89)
- MM – Management Module
- SFM – Switch Fabric Modules
- Compact-Flash – Supposedly where the firmware/configs are stored.
- CID-Card – Chassis ID – each card has two EEPROM – one critical and a non-critical. The non-critical can be fixed with a “CID Recovery Tool”
oscmd – more details about this, how to run a command? (p96)
- oscmd arp -a
- oscmd scp localfile remote.server:

The below I’ll bring up in a later post:

VDX Troubleshooting Course

BCEFP practice questions / answers

http://community.brocade.com/t5/Certification/BCEFP-2013-Exam-150-180-Practice-Questions/ta-p/4099

Other

Intro to VCS Fabric Technology: http://www.brocade.com/downloads/documents/white_papers/intro-vcs-fabric-technology-wp.pdf
CFP- MSA CFP2 Hardweare Specs: About the 40/100Gbps CFP2 SFP. MSA – multi-source agreement.
Code names of switches? Find the NOS firmware and look in the file “platform_names”. Quite a few bird names (nighthawk, dragon, superhawk, tomahawk ;), kestrel, falcon, blackbird).

Brocade Certified Ethernet Fabric Professional 2015 Beta Exam

2 Replies

Contents

1 Intro
2 Objectives for 2013 exam
3 Page numbers for 2015 beta exam below:

Intro

http://community.brocade.com/t5/Certification/BCEFP-2015-Beta-Exam-Information-and-Study-Material/ta-p/58276

The course materials, including references to various resources such as the NOS Admin guide are available on the page above.

The Advanced Ethernet Fabric Troubleshooting (VDX-TS 300-WBT) has the pdf’s on the link above, but it’s also currently free on brocade’s saba education page.

Objectives for 2013 exam

Objectives for the exam (2013 version, so might be different for 2015) are:

Theory and Concepts

Describe the VCS implementations of TRILL
Describe the rate-limiting features in a VCS fabric
Identify basic routing concepts and how they interact with an Ethernet fabric
Identify VDX hardware components support.

Design

Describe the benefits of using TRILL
Describe QoS in a VCS fabric
Demonstrate knowledge of various types of link aggregation in a VCS fabric
Describe VDX hardware used in the design of a VCS fabric
Describe AMPP concepts

Implementation and Configuration

Demonstrate knowledge of sharing native FC storage with FCoE devices in a VCS fabric
Describe the implementation of lossless Ethernet for FCoE and iSCSI traffic
Describe how to integrate AMPP into a vCenter environment
Demonstrate knowledge how to implement Layer2/Layer3 ACLs in a VCS fabric
Demonstrate knowledge how to configure VRRP/VRRP-E on a VDX
Demonstrate knowledge how to configure a VCS fabric to connect to traditional Layer 2/Layer 3 switches
Demonstrate knowledge how to implement vLAGs

Management

Demonstrate knowledge of VDX management features

Troubleshooting

Demonstrate advanced troubleshooting knowledge
Demonstrate knowledge how to troubleshoot native FCoE and VCS to FC SAN bridging
Demonstrate knowledge how to troubleshoot VCS to an IP network

Page numbers for 2015 beta exam below:

Network OS Administrator’s Guide v4.1.1 53-1003225-01

Pages 11-20,26-31,54-56,75-77,94,115,119,135-144,149,150,191,201-203,210,243-247,253,261,269,311-314,321,

322,338-345,361-385,400,403,406,425-433,437,465,483,495,504,506,517-521,541,559,563,565,583,593,594,597,

603-609,661-663,668,676,682,686,714,715

Network OS Command Reference v4.1.1 53-1003226-01

Pages 299, 1258-1260,1266,1297,1317,1318

Network OS v4.1.1 Brocade VDX Release Notes

Pages 4,10,28,50

Network OS Software Licensing Guide v4.1 53-1003164-01

Pages 11-13

VDX 6740 Hardware Reference Manual 53-1002829-02

Page 1
VDX 8770-4 Hardware Reference Manual 53-1002563-03

Chapter 1, Page 1; Chapter 3, Page 32

VDX 8770-8 Hardware Reference Manual 53-1002564-03

Chapter 1, Page 1
VDX 6730 Hardware Reference Manual 53-1002389-06

Pages 1,2,15

Brocade VDX 8770 Switch Data Sheet GA-DS-1701-04

CFP2 Hardware Specification Draft Revision 0.3

Page 46