Tag Archives: kernel

Setup a 3 Node Lustre Filesystem

Introduction

Lustre is a filesystem often used by clusters because many computers can mount the filesystem simultaneously.

This is a small log/instruction for how to setup Lustre in 3 virtualized machines (one metadata server, one object storage server and one client).

Basic components:

VMWare Workstation
3 x CentOS 6.3 VMs.
Latest Lustre from Whamcloud

To use Lustre your kernel needs to support it. There’s a special one for server and one for the client. Some packages are needed on both.

Besides lustre you’ll need an updated version of e2fsprogs as well (because the version that comes from RHEL6.3 does not support large partitions).

Starting with the MDS. When the basic OS setup is done will make a copy of that to use for OSS and Client.

Setup basic services.

Install an MDS

This will run the MDT – the metadata target.

2GB RAM, 10GB disk, bridged networking, 500GB for /boot, rest for / (watch out, it may create a really large swap). Minimal install. Setup OS networking (static ip for servers, start on boot, open port 988 in firewall, possibly some for outgoing if you decide to restrain that too), run yum update and setup ntp. Download latest lustre and e2fsprogs to /root/lustre-client, lustre-server and e2fsprogs appropriately (x86_64). Lustre also does not support selinux, so disable that (works fine with it in enforcing until time to create mds/mdt, also fine with permissive until it’s time to mount).
Put all hostnames into /etc/hosts.
Poweroff and make two full clones.
Set hostname.

Install an OSS

This will contain the OST (object storage target). This is where the data will be stored.

Networking may not work (maybe device name changed to eth1 or eth2).
You may want to change this afterwards to get the interface back to be called (eth0). A blog post about doing that.

Install a client

This will access and use the filesystem.

Clone of the OSS before installing any lustre services or kernels.

Install Lustre

Before you do this it may be wise to take a snapshot of each server. In case you screw the VM up you can then go back :)

Starting with the MDS.

Installing e2fsprogs, kernel and lustre-modules.

Skipping debuginfo and devel packages, installing all the rest.

yum localinstall \ 
kernel-2.6.32-220.4.2.el6_lustre.x86_64.rpm kernel-firmware-2.6.32-220.4.2.el6_lustre.x86_64.rpm \
kernel-headers-2.6.32-220.4.2.el6_lustre.x86_64.rpm \
lustre-2.2.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64.rpm \ 
lustre-ldiskfs-3.3.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64.rpm \
lustre-modules-2.2.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64.rpm

The above was not the order they were installed. Yum changed the order so that for example kernel-headers was last.

yum localinstall e2fsprogs-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-debuginfo-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-devel-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-libs-1.42.3.wc3-7.el6.x86_64.rpm \
e2fsprogs-static-1.42.3.wc3-7.el6.x86_64.rpm \
libcom_err-1.42.3.wc3-7.el6.x86_64.rpm \
libcom_err-devel-1.42.3.wc3-7.el6.x86_64.rpm \
libss-1.42.3.wc3-7.el6.x86_64.rpm \
libss-devel-1.42.3.wc3-7.el6.x86_64.rpm

After boot, confirm that you have lustre kernel installed by typing:

uname -av

and

mkfs.lustre --help

to see if you have that and

rpm -qa 'e2fs*'

to see if that was installed properly too.

By the way, you probably want to run this to exclude automatic yum kernel updates:

echo "exclude=kernel*" >> /etc/yum.conf

After install and reboot into new kernel it’s time to modprobe lustre, start creating MDT, OST and then mount things!
But hold on to your horses, first we ned to install the client :)

 

And then the Client

Install the e2fsprogs*

We cannot just install the lustre-client packages, because we run a different kernel than the ones that whamcloud have compiled the lustre-client against.

We can either back-pedal and install an older kernel. Or we can build (from source / SRPMS) a lustre-client that works on a kernel of our choosing. The later option seems like a better way, because we can then upgrade the kernel if we want to.

 

Build custom linux-client rpms

Because of a bug it appears that some ext4 source packages are needed – while they are not. You need to add some parameters to ./configure. This will be the topic of a future post.

The above rpmbuild should create rpms for the running kernel. If you want to create rpms for a non-running kernel you are supposed to be able to run.

Configure Lustre

Whamcloud have good instructions. Don’t be afraid to check out their wiki or use google.

/var/log/messages is the place to look for more detailed errors.

On the MDS

Because we do not have infiniband you want to change the parameters slightly for lnet to include tcp(eth0). These changes are not reflected until reboot (quite possibly something else) – but just editing a file under /etc/modprobe.d/ called for example lustre.conf is not enough.

Added a 5GB disk to the mds.

fdisk -cu /dev/sdb; n, p, 1, (first-last)

modprobe lustre lnet

mkfs.lustre –mdt –mgs

mount

On the OSS

Also add the parameters into modprobe.

mkfs.lustre –ost

mount

On the client

Add things into modprobe.

mount!

Write something.

Then hit: lfs df -h

To see usage!

 

Get it all working on boot

You want to start the MDS, then the OSS and last the client.
But while it’s running you can restart any node and eventually it will start working again.

Fstab on the client:
ip@tcp:/fsname /mnt lustre defaults,_netdev 0 0

Fstab on the OSS and MDS:
/dev/sdb1 /mnt/MDS lustre defaults,_netdev 0 0

While it’s running you can restart any node and eventually it will start working again.

Upgrade Ubuntu 10.10 to Natty

Saw this note today in the MOTD – that I can upgrade with do-release-upgrade.

Official instructions/details from Ubuntu and here is another quality post  on unixtutorial.

My Ubuntu 10.10 is running inside a VMWare Workstation Virtual Machine on my Windows 7 x64.

I did it over ssh with ‘sudo screen -S upgrade do-release-upgrade‘ .
Probably not wise to do this over ssh, but the setup started an extra sshd service on another port, just in case something breaks.

It would need to download about 210MB of packages, if you press on ‘d’ at the right time you’ll get into a ‘less’ of all the removes, upgrades and new installs. Press q to exit that.

During upgrade:

  • I got to chose keyboard (I have an IBM Thinkpad T40, but it wasn’t in the list, took an R60 instead, looked pretty similar).
  • It then asks for which services that are using NSS (I had no idea, so just used the default of rsync, mysql, apache and one more).
  • memtest86?? Does it come with this automagically? This is pretty cool, I’ve only heard about this being used in an iso to quite thoroughly test memory. The OS uses quite a lot of the RAM so if you test RAM from inside the OS you will not be able to test all, thus the boot CD. However, later on it turns out that memtest86 is actually put in the grub/boot menu! Very handy!
  • apparmor – this is a security module – apparently you can give applications profiles
  • 2.6.38! woop! See the link for more updates in the kernels. On that page – kernelnewbies.org – you can also find details about the other updates. Kernel.org only has detailed change log as far as I could see (lots of text).
  • Later on it asks you, do you want to delete these packages (d for details): libisc60, libdns66, linux-image-2.6.35-22-generic, python-newt, libxapian15, byobu
  • Last question: do you want to reboot? – No pain, no gain. Is ‘adventurous’ to do kernel upgrades remotely ;)
  • VM came back online after what felt like just 15 seconds.
  • Welcome to Ubuntu 11.04 (GNU/Linux 2.6.38-8-generic x86_x64)!

Red Hat Enterprise Linux(RHEL) in VMWare Workstation

Test with Red Hat Enterprise Linux (RHEL).

Download: Sign up for an evaulation on https://access.redhat.com/downloads/

rhel-server-5.6-x86_64-disc1.iso

VMWare Workstation does find this in “easy install”. Not doing that this time.

20GBdisk (default) and 1552MB RAM (default 1024MB)

  1. install in either graphical or text mode, going with graphical. There are also special modes. Maybe something similar to Ubuntu’s minimal virtual kernel is available?
  2. test cd
  3. Mouse works!
  4. Get subscription number with the help of this: https://access.redhat.com/kb/docs/DOC-15404. Copy paste did not work.
  5. Filesystem stuff. Modify or not. Encrypt or not. I went with default and encryption. For encryption you need to set a boot password (min 8 chars).
  6. IP/Timezone settings.
  7. root password (not min 8 chars)
  8. software sets – software development, virtualization, web server. I went with the two last. You can also customize it deeper. Like: gnome/kde? Printing support? Samba? I chose web server but mysql was not selected, not the php-mysql plugin for apache either. Virtualization is Xen – openfabrics enterprise distribution for RDMA/infiniband stuff.
  9. cool stuff found: iptraf, hwbrowser, vnc
  10. /root/install.log for install .. log.
  11. After this it says that it will require all cds.. but I want to download them instead. How? proceeding anyway, let’s see what happens. Maybe it gives the opportunity to download instead. Googling in the meantime. Doesn’t look good. One way to do it would be to put the CDS/dvd on a network/http server in your LAN. But it does not mention a public repository etc.
  12. Formatting, then installing. It asks for CD2. No other buttons. Getting DVD instead. rhel-server-5.6-x86_64-dvd.iso
  13. DVD went fine, nothing after this, just reboot.

Booting

  1. Insert LUKS password – the encryption password you entered before.
  2. IPv6 failed during first boot.
  3. Also some kind of disk monitoring.
  4. Then a little configuration! This is nice. In ubuntu/debian it just goes into the system with a bunch of default setings.
  5. Like firewall, enabled/disabled. Trusted srevices.
  6. SELinux – ‘improved’ security controls, enforced/permissive/disabled. Keeping default: enforced.
  7. no kdump
  8. NTP! Enabling this, using default ntp servers (0.rhel.pool.ntp.org) and disabling ‘use local time source’. This part contacts the NTP server during install, which worked, so that looks good.
  9. Connect to RHN. Said yes. Takes a long time to register? no contact. Trying this later.
  10. Set up a new user. You can use kerberos or NIS too from here.
  11. Insert additional CDS
  12. Login prompt!

After login

  1. VM -> Install VMWare Tools – I want to use “Unity” in Red Hat. I’ve used it for Windows XP (had a guest os for work) and it was great.
  2. Right-click the tarball and ‘extract to’. You need to have root access when you run it. So open a terminal and type ‘su -‘ – this will give you the root prompt.
  3. Then go to where you extracted it. ./vmware-install.pl.
  4. Gives message that I apparently am running a Xen kernel and that this is not supported. Trying anyway. Answering yes as default on the questions.
  5. Install was successful, opening configuration tool. Some green ‘OK’ s.
  6. Before we can compile we need to have make and gcc installed. It also asks for kernel headers that it couldn’t find. Going with the default “” on that.
  7. memory manager, vmhgfs (filesystem driver for shared folders), vmxnet (fast ethernet), vmblock (drag ‘n’ drop), communication service, vsock, vmxnet3 (virtual network card), pvscsi,  – not installed because no compilation software like make/gcc installed.
  8. x configuration, host resolution found but vm resolutions max at 800×600!?
  9. restarting and it said good stuff, but unity does not work and resolution cannot be changed above 800×600.
  10. accessing via ssh works fine too

Ubuntu 10.10 + VMWare + Irssi

How small VM can you make if you are only going to use it to run Irssi in a screen?

OS: Ubuntu 10.10 x64 Virtual Kernel
Hypervisor: VMWare Workstation

Disk – no logs – 1.10GB is what my previous took, with samba, so probably less but 1.1 should be all right, don’t want it to run out of space either. Should probably partition /var/log into its own so that if that fills up (maybe after bruteforce ssh logins) then it doesn’t fill up the rest of /.
RAM –

  • turn off cron jobs
  • install virtual kernel

about the cron jobs, I just installed a ubuntu virtual kernel and only cron job running (as seen in syslog) is this:

CRON[9141]: (root) CMD (   cd / && run-parts –report /etc/cron.hourly)

Test 1

1.2GB disk
64MB RAM

Kernel panic – not syncing: attempted to kill init!
ctrl-alt-del in the VM doesn’t work, had to hard kill it :p
Reboot – same problem. It does this after selecting minimal virtual kernel and pressing install Ubuntu Server.

Test2

Increased RAM to 96MB and now it passed.
However, after selecting keyboard it still crashes. So 96 is also too little.

Googled around a little and found an article on Ubuntu.com that gives some insight.

For example: if you run out of memory, then it will swap. So if you are not running out of memory then it’s probably better to give it a little more to be on the safe side.

Test 3

Increased RAM to 128MB
Now it goes even longer.

Partitioning -> chose manual.
There is a device there ‘sda’ -> SCSI3. Mark that and hit enter. Say yes to create new empty partition. Then it creates a new space pri/log. Create new partition.

As size, type in: 100MB
Primary, beginning, default settings on the filesystem etc but mount point: /var/log. Then done setting up this partition.

Then we will create another partition for / (or the rest).

Like this:

ubuntu_partition right

ubuntu_partition right

The above does not include a swap space, and the installation complains that there is no swap space defined and that there may be problems if this is not configured during install.

Let’s see :)

** maybe Ubuntu is not the slimmest OS to install for this purpose. The guide I linked to above mentions a DSL – damn small linux. But we want a minimal server OS, not a desktop one. Maybe the new Debian 6 would be cool to try.

During install there was a dialogue about something being already on the disk.. and that this could cause issues. Maybe this was copied there from one of the previous tries with less amount of RAM. Went back and erazed this on each just to be on the safe side. Very slow though. About 1% / s. After this I went into each and set them to format instead. It mentions old installation files anyway. Proceeding. This happened twice? Three times now.. Maybe this is not going so well. Ok happened four times.

Hit Alt+F2 (and then enter to get the console) and then df -h. Nothing is over 20% except the SR0/cd-rom which is at 100%. Did a ‘more /var/log/syslog’ and at the end there are some ‘Out of memory’ things going on.

So there we go. 128MB is too little.

But here:  it says that 128MB is the requirement..

Test 4

I will try to install again and not mess up with the partitions.
1.2GB and 128MB again.
Going pretty well, looks like it’s stopped at 75% and “storing language”. Patience. Ok, 10mins later still there. Nothing about out of memory in /var/syslog. The Guest was behaving a little slow though when working the console.
Left it on over night and when I looked again the following day it was at the next step!

Installing openssh via the installation menu this time.
Taking quite long time this time as well. Like hours. 2 hours now. Nothing happening in syslog. Think I’ll give Debian a shot next Test.

In the meantime did Test 5. However 4 hours later, still configuring ‘language-pack-en-base’.

Now, approximately lots of hours later. It is at configure grub.

And yay, it finally boots!

user@irssi:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             1.1G  430M  606M  42% /
none                   54M  164K   54M   1% /dev
none                   57M  4.0K   57M   1% /dev/shm
none                   57M   36K   57M   1% /var/run
none                   57M     0   57M   0% /var/lock
none                  1.1G  430M  606M  42% /var/lib/ureadahead/debugfs
/home/martbhell/.Private 1.1G  430M  606M  42% /home/martbhell

And the size of the folder in Windows: 750 MB (787 341 312 bytes)

With 128MB ram there is maybe 10MB free and it swaps a little (just a few kB so far).

Test 5

debian-6.0.0-amd64-netinst
Using Debian 5 64-bit.

  1. graphical menu is seen, no advanced options used
  2. chose languages – look a lot like Ubuntu but there are some differences:
  3. After choosing a hostname it asks for domain. Put in WORKGROUP.
  4. Set a root account password (so no more sudoing – probably just add my user to the wheel/root group).
  5. and you have a few more options in the partitioning, like a separate /home partition
  6. then you get to chose a debian mirror (becuase I use the netinst). ftp.fi.debian.org is the one I chose
  7. you can participate in a “most used packages” survey
  8. software selection: graphical desktop, web, dns, ssh, laptop, standard system utilities. I chose SSH and standard system utilities.
  9. grub, then reboot and loading!

Internet works fine from the start. apt-get update; apt-get install irssi
Remember, here you have to log in as root to run ifconfig or apt-get.

It’s the same way in Debian as in Ubuntu to set static ip. Just edit /etc/network/interfaces / don’t forget you can just restart the networking services by ‘/etc/init.d/networking start’ instead of rebooting ;)
Now, I actually forgot to set minimal ram/disk for this one. So we have to do this again ;)

Test 6

debian-6.0.0-amd64-netinst
Using Debian 5 64-bit.
128MB RAM

One thing that’s cool about a VM is that you can resize the amount of RAM whenever (probably good to turn off the guest first). So how about just lowering it instead of installing a new one?

OK, so it now has 512MB. Going down to 64 in one go (listed as minimum in VMWare Workstation).

It’s swapping after just a few minutes with screen+irssi.

to sort by memory usage in top press SHIFT+m

biggest memory hogs (all over 1% – figures in bold) are :

1474 user 20   0 23388 6220 1572 S  0.0 11.2 0:00.25 bash
1503 user 20   0 50084 5228 3824 S  0.0  9.4 0:00.06 irssi
1470 root      20   0 70488 3280 2584 S  0.0  5.9 0:00.03 sshd
913 root      20   0  117m 1788  904 S  0.0  3.2 0:00.01 rsyslogd
1473 user 20   0 70488 1680  964 S  0.0  3.0 0:00.06 sshd
1502 user 20   0 25184 1472  992 S  0.0  2.6 0:00.01 screen
1546 user 20   0 19040 1300 1004 R 99.9  2.3 0:00.01 top
985 root      20   0 22392  712  512 S  0.0  1.3 0:00.00 cron
1233 Debian-e  20   0 44140  660  392 S  0.0  1.2 0:00.00 exim4
1472 root      20   0  5928  620  520 S  0.0  1.1 0:00.00 getty
1 root      20   0  8352  616  560 S  0.0  1.1 0:01.42 init
1277 root      20   0 49168  544  428 S  0.0  1.0 0:00.00 sshd

What I might be able to get rid of is rsyslogd and cron. But then again, if I were to connect this to the internet so I could access it and resume the screen/irssi from anywhere, I would want to keep track of what is happening on the machine.

user@debian:/var/log$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             3.8G  638M  3.0G  18% /
tmpfs                  28M     0   28M   0% /lib/init/rw
udev                   23M  140K   23M   1% /dev
tmpfs                  28M     0   28M   0% /dev/shm

Windows usage: 887 MB (930 816 000 bytes)

Summary

The Ubuntu Server 10.10 with the minimal virtual kernel took forever to install (maybe it would have been faster to have more mem during install and then lower when it’s done) and with 128MB it still swaps a little with only screen and irssi running. But it does use about 130MB or 200MB less space than the Debian6 guest.

The Debian 6 however runs OK with 64MB, swaps a little at that though so I would probably run this with 96 or 128MB just to be on the safe side if I were to run it.

File share from Ubuntu 10.10 with Windows 7 Client

Figured I would give this a shot and see how this is done in Linux.

Overview

1x Ubuntu 10.10 VM in VMWare Workstation. Installed with virtual kernel.
1x Windows 7 VM. All updates.

Not going to go through the installations in this post, just the domain/LDAP part. See previous posts for installation stuffs.

Found this article on ubuntu.com -> -> 10.10/serverguide/C/network-authentication.html .. But this does not exist anymore.

Whoa, quite a lot to do. Think I will read the manuals this time :)

Basically I think I just need to install and then configure Samba.
Because it has file-sharing and authentication/authorization.

So, first step will be to install Samba and try file-sharing.

File Sharing

sudo apt-get install samba

installs these:

The following NEW packages will be installed:
libavahi-client3 libavahi-common-data libavahi-common3 libcups2
libfile-copy-recursive-perl libgnutls26 libldap-2.4-2 libsasl2-2
libsasl2-modules libtalloc2 libtasn1-3 libwbclient0 samba samba-common
samba-common-bin update-inetd

After that I can run \\192.168.0.ip which is the IP of the VM running samba – it gives me a login prompt.

Uncommenting this in /etc/samba/smb.conf

[homes] comment = Home Directories
browseable = yes

Still asks for a password. Tried with user@ip – this seems to work. It shows a directory called “homes” but it doesn’t work to browse into it.

[2011/02/06 16:30:45.949726,  1] smbd/service.c:678(make_connection_snum)
create_connection_server_info failed: NT_STATUS_ACCESS_DENIED

Set ‘share’ and set the guest account = nobody  then I got this:

[2011/02/07 13:22:06.770082,  0] smbd/service.c:988(make_connection_snum)
canonicalize_connect_path failed for service foo, path /mnt/foo

Then what I did was this:

  1. created a directory called /samba
  2. sudo chmod +x /samba
  3. sudo chmod 777 /samba
  4. sudo chown nobody /samba
  5. add this to /etc/samba/smb.conf
  6. [foo] comment = foo
    path = /samba
    read only = no
    guest ok = yes
    guest only = yes
    browseable = yes
  7. security = share
  8. guest account = nobody

And then \\ip\foo and woopsie! I can both write and read :)

[2011/02/07 13:23:14.022980,  1] smbd/service.c:1070(make_connection_snum)
192.168.0.ip (192.168.0.ip) connect to service foo initially as user nobod

yay!

Ubuntu 10.10 Minimal Virtual Kernel + VMWare Workstation

To install Ubuntu 10.10 with a virtual kernel instead of the normal one = good, less stuff installed that you may not need.

  1. When setting up the install, do not use the easy install. Chose to install an OS later. Set up bridged/nat depending on which one you want.
  2. Add the install .iso to the CD-drive in the VM
  3. Select a language
  4. Press F4 (it didn’t work in the first screen)
  5. Chose – install a minimal virtual machine
  6. Install Ubuntu Server
  7. Chose language again
  8. Chose key map – (I chose English and had to browse to Finland)
  9. Asked to press some buttons, wanted Swedish (but have an English keyboard) so tried to press the right ones :p
  10. Then time zone Helsinki/Finland was found.
  11. Using default (whole disk, no encryption or lvm) for partitioning.
  12. set up users
  13. set up encryption on home dir
  14. proxy setup
  15. installing security updates automagically
  16. any extra packages (DNS, LAMP, Mail, OpenSSH, etc)? – I chose no, want to chose this myself later.
  17. yes I want grub (it finds only one OS on the virtual disk ;)

Then I see the login prompt! Obviously the easy-install in VMWare Workstation has a lot less steps :)

But on the other hand you could install OpenSSH directly through the install and then you do not have to log on to the VM via VMWare Workstation, but can do it via your favorite ssh program instead.

Post install

What I want installed every time after an uninstall.
After install it is a very very small installation.
Not even ‘man’ is installed.

sudo apt-get install openssh-server ntp nano

edit /etc/network/interfaces – configure static ip
edit /etc/ntp.conf – add time servers
edit ~/.bashrc – change colors in the prompt and add color

Kernel difference you can see when running uname: 2.6.35-22-virtual in comparison to 2.6.35-22-generic

There!

Now you can set up whatever you want on it! Of course you may want to do more things, set up iptables or you could use it like it is before the things I do after each install. You can use vi instead of nano/pico and use dhcp instead, depends on what you are going to do with your VM.