Tag Archives: bash

Update to Spotify – An RSS Feed

After some time the solution I devised on http://www.guldmyr.com/blog/script-to-check-for-an-update-on-a-web-page/ just did not elegant enough (also it stopped working).

Instead of getting some kind of output in a terminal sometimes somewhere I decided to make an RSS feed that updates http://guldmyr.com/spotify/spot.xml instead :)

I suspect that the repository itself could be used to see if there’s an update to it. It has all these nice looking files in here: http://repository.spotify.com/dists/stable/ – but I also suspect this is a repository for debian/ubuntu which I cannot use on my RHEL-based workstation.

Thus:

A bash script was written. It uploads the spot.xml whenever there is an update. The script does not run on the web-server so it ftps the file to the web-server, it would be nice if it did because then the actual updating of the feed would be so much more simple (just move/copy a file).

But, I hope it works :) Guess we’ll see next time there’s an update to spotify!

The script itself is a bit long and I hope not too badly documented, so it’s available in the link below: http://guldmyr.com/spotify/update.spotify.rss.feed.sh

Or, more easily, you can just add http://guldmyr.com/spotify/spot.xml to your RSS reader (google’s reader, mozilla’s thunderbird, there are many of them).

Some things I learned:

  • Latest post in an RSS feed is just below the header, making it a bit awkward to update via a script as you cannot just remove the </channel> and </rss>, add a new <item></item> and then add the </channel> and </rss> at the end again.
  • lastBuildDate in the header also needs to be updated each time the feed is updated. In the end I decided to re-create the file/feed completely every time there was an update.
  • Some rss-readers appear to have a built-in interval that they use to check if there’s an update. So for example you could update the rss-feed and press ‘refresh’ but the client still won’t show the new feeds. Google Reader does this for example. With Mozilla’s Thunderbird you can ask it to update (Get Messages) and it will. You don’t need an e-mail account in Thunderbird to use it as an RSS reader by the way.
  • http://feedvalidator.org is a great tool, use it.

I claim no responsibility if you actually use the script, the feed however should be fairly safe to subscribe to.

 

[Valid RSS]

Install FreeBSD in VirtualBox

The Past

I used to run FreeBSD 5 and 6 about eight years ago on a Pentium III 900MHz machine with maybe 768MB RAM. It was very slow but after a lot of tinkering with the kernel I got it to boot and run very nicely.

Fluxbox was the window manager I used then together with Eterm and pico :)

The Install

Installing it $today in a VirtualBox 4.1.6 on an IBM T40 running RHEL 6 x64.

I used the ‘disc1’ .iso of FreeBSD 8.2. Give it enough of RAM and bridged networking is probably what you want. If you have it set to NAT and then want to change you can do that ‘online’ while the virtual machine is online. Might be good to run another dhcp discover after though. And also don’t forget to remove default route before that. (route del default).

Defaults are pretty OK I suppose on a VM. No need to mess around with the partitions or labels.

It’s all done in the console menus and you can go back and forth between the menus. You probably want to enter the ‘post-install’ section to set ip, password, chose to install sshd. Add a new user and add it to the group ‘wheel’. That way you can hit ‘su -‘ to get root access.

Using FreeBSD

After it’s up you probably want to get root access: ‘su -‘
Run ‘dhclient INTERFACE’ to get a dhcp ip. Find the interface name via ifconfig.

Install Bash

If you chose to install Ports during install, you can go to /usr/ports and hit ‘make search name=”bash”. You can use this to search for packages called bash. Then cd /usr/ports/shells/bash; make; make install; make clean

Then ‘chsh’ and change to /usr/local/bin/bash. Vi-syntax works so press i to insert, r to replace, x to remove or :wq! to write and quit.

Update and use ports.

You can also search on freshports.org.

csup is a tool that you use to update the ports collection. Another is portsnap.
Portsnap appears to be a bit simpler and it came by default when I installed it.

portsnap fetch; portsnap extract; port portsnap update

Basically Ports consists of softwares’ make files and some files pointing where to download the softwares when you decide to install them. You only need to run the portsnap extract the first time. Rest of the times fetch and update should do it. Or ‘portsnap cron’ if you do it in the scheduler/crontab.

Portaudit is a tool that checks for vulnerabilities in the softwares you use. This was very quick to install though. To see if you have any issues, hit ‘portaudit -Fda’.

Each port or software should have a file called pkg-descr, you can use this to see what it does.

portsearch is one that you can use to search for ports instead of the ‘make search’.

sshd

You can ssh into the machine directly after installing, as long as it has IP connectivity (if you can ping it). You also cannot ssh in as root by default.

firewalling

the firewall is ‘ipfw’. This is not enabled by default. To enable it set firewall_enable to YES in /etc/rc.conf. Make sure you add some good fw rules first. Or you can set firewall_type=”open” in rc.conf and then firewall_script=”/etc/ipfw.rules”.

See http://www.freebsd.org/doc/handbook/firewalls-ipfw.html . There is a sample called ‘inclusive ruleset’. This one you can paste into /etc/ipfw.rules , edit to your liking (change your public interface name, add dns-servers, comment out services you don’t need (like port 80 if you for example do not have a web-server). You could then edit this script to have a

$cmd 00411 allow tcp from 192.168.0.0/24 to me 22 in via $pif setup limit src-addr 2

This would allow only addresses from the 192.168.0.0/24 network to ssh into your machine if you comment the rule that allows incoming on port 22 from anywhere.

ipfw list # to see the current firewall

Conclusion

FreeBSD is special compared to a few other operating systems because you get to compile all the software. You can of course get binaries if you want and install via pkg_add. But that’s not so cool right? It’s also a good idea to tweak the kernel, especially if you have a little slower system and want some better performance. If you have a slower system (like in a virtual machine), it could be painfully slow to install something. For example bash took what felt like forever to install for me.

This means a bit more patience is required with FreeBSD, but on the other hand maybe this way there will much be less crap installed.

Apparently FreeBSD 8.2 is not so cool because there is a 9 in beta or PC-BSD. If you want you can even get a ‘snapshot’ in the CURRENT subset, which is basically as new as it gets.

Simple RRD graphs

This is how to create simple RRD graphs using one data source that can be 0 and above. It is not an “ever increasing” counter.

It will look like this:

example rrd graph

 

1. Create the rrd database

I wrote this down in a .sh file so I can go back later and see how it was set up.

#!/bin/sh
rrdfile=’/home/$user/rrd/movers.rrd’
rrdtool=’/usr/bin/rrdtool’
$rrdtool create $rrdfile –step 300 DS:movers:GAUGE:600:U:U RRA:AVERAGE:0.5:1:576 RRA:AVERAGE:0.5:6:672 RRA:AVERAGE:0.5:24:732 RRA:AVERAGE:0.5:144:1460

#5 minute step (base interval with which data will be fed into the RRD)
#10 minute heartbeat for the data source
#2 days of 5 minute averages
#2 weeks of 1/2 hour averages
#2 months of 2 hour averages
#2 years of 12 hour averages
 

2 Add data to the rrd

Also done in a bash script. Because the –step above is set to 300, you need to run this script every 300 seconds (or every 5 minutes). The script specified in $allpools prints the path to a file. Then with $output and $movers that file is grepped for ‘RUNNING’ and then it counts how many lines that was – amount of movers.

#!/bin/sh
rrdfile='/home/$user/rrd/movers.rrd'
rrdtool='/usr/bin/rrdtool'

allpools='/home/$user/bash_script.sh'
output=$($allpools)
movers=$(grep RUNNING $output|wc -l)

$rrdtool update $rrdfile N:$movers

The N: is NOW. $movers is the value you want to plot.

3. Make a graph

Add this to another .sh script. This you can run at whatever interval you want.

#!/bin/sh
rrdfile='movers.rrd'
rrdtool='/usr/bin/rrdtool'
dpath="/var/www/html/graphs/movers"
alltimeimage="$dpath/movers.png"
lastweekimage="$dpath/moverslw.png"
lastmonthimage="$dpath/moverslm.png"
last4hoursimage="$dpath/moversl4h.png"
last3monthsimage="$dpath/moversl3m.png"
lastday="$dpath/moversld.png"
enddate=$(date +%s)
#enddate is the same as "now"
cd /home/$user/rrd

$rrdtool graph $alltimeimage --end now --start 1321603000 \
        -v from_beginning -t active_movers \
        DEF:movers=$rrdfile:movers:AVERAGE LINE:movers#000000
#rrdtool graph /path/to/image.png --end now --start when_I_started_capturing -v label_left -t title_top \
#DEF: as I only have one I only used movers, maybe you can change the names in case you have several data sources
#DEF: you can also use other things than AVERAGE (like MIN/MAX)
#LINE: #000000 is black

$rrdtool graph $lastweekimage --start -1w \
        -v last_week -t active_movers \
        DEF:movers=$rrdfile:movers:AVERAGE LINE:movers#000000 \
        AREA:movers#8C2E64 \
        GPRINT:movers:LAST:"Current\: %1.0lf" \
        GPRINT:movers:MAX:"Max\: %1.0lf" \
        GPRINT:movers:MIN:"Min\: %1.0lf" \
        GPRINT:movers:AVERAGE:"Avg\: %1.0lf"

#If you want to make it a little more complex. AREA fills the space between the value and the x-axis.
#GPRINT statements print some values relating to the graph.

$rrdtool graph $last4hoursimage --end now --start end-4h \
        -v last_4_hours -t active_movers \
        DEF:movers=$rrdfile:movers:AVERAGE LINE:movers#000000

$rrdtool graph $lastmonthimage --end now --start -1m \
        -v last_month -t active_movers \
        DEF:movers=$rrdfile:movers:AVERAGE LINE:movers#000000

$rrdtool graph $last3monthsimage --end now --start -8035200 \
        -v last_3_months -t active_movers \
        DEF:movers=$rrdfile:movers:AVERAGE LINE:movers#000000

$rrdtool graph $lastday --end now --start -1d \
        -v last_day -t active_movers \
        DEF:movers=$rrdfile:movers:AVERAGE LINE:movers#000000

4. Crontab – scheduling

*/5 * * * * /bin/bash /home/$user/rrd/rrd.update.sh > /dev/null 2>&1
#every 5 minutes
*/15 * * * * /bin/bash /home/$user/rrd/rrd.graph.sh > /dev/null 2>&1
#every 15 minutes

 

5. Final Words

I am not providing the data gathering script here as you probably won’t need it: it lists movers (transfers) on all pools in a dCache system.

Script To Check For an Update on a Web Page

Hey!

This is used for me on my Linux workstation to get a notification if there is a new spotify release whenever I open a new terminal.. It would be applicable for other (probably also simple) pages that aren’t updated frequently.

Reason: http://repository.spotify.com/pool/non-free/s/spotify/

I wanted to see if there was a new spotify release for Linux/QT.

Method: The URL is above – but what if I do not want to go there every day and get disappointed?

Way nicer to have a script do it for me.

This script saves the index.html from the URL above of each day.

Then each day when it downloads the .html it checks if it’s different from yesterday.

This has its limitations, if there is an update in the weekend I will never know.

The script should check the last x amount of days and if any of them are different from today it should tell me. The script checks if any of the files are different from today, if so, it will write something into another file. The script then checks if this file is non-empty, if it has data in it, it will write to this other file that. Tada. :p

If it is, then it will write to a file that is referenced in $HOME/.bashrc.

The layout of the blog doesn’t like really long lines in <pre>, but you can select below and only get the post (and not the stuff on the right side).

spot_check.sh:

#!/bin/sh

dat1=$(date +%Y.%m.%d)
daty=$(perl -MPOSIX=strftime -le 'print strftime "%Y.%m.%d",localtime (time - 86400)')
dat2=$(perl -MPOSIX=strftime -le 'print strftime "%Y.%m.%d",localtime (time - 172800)')
dat3=$(perl -MPOSIX=strftime -le 'print strftime "%Y.%m.%d",localtime (time - 259200)')
dat4=$(perl -MPOSIX=strftime -le 'print strftime "%Y.%m.%d",localtime (time - 345600)')
dat5=$(perl -MPOSIX=strftime -le 'print strftime "%Y.%m.%d",localtime (time - 432000)')

path="$HOME/Downloads/Spotify/saved"
out="$HOME/Downloads/Spotify/diff.log"
bout="$HOME/.spotcheck"
wget -q http://repository.spotify.com/pool/non-free/s/spotify/ -O $path/$dat1.html

diff -q $path/$dat1.html $path/$daty.html > $out
diff -q $path/$dat1.html $path/$dat2.html >> $out
diff -q $path/$dat1.html $path/$dat3.html >> $out
diff -q $path/$dat1.html $path/$dat4.html >> $out
diff -q $path/$dat1.html $path/$dat5.html >> $out

if [[ -s $out ]] ; then
echo $out "is not empty";
echo "#!/bin/sh" > $bout;
echo "echo new spotify release" >> $bout;
chmod +x $bout;
else
echo $out "is empty";
echo "No new spotify release.";
rm $bout;
fi;

Crontab (daily at 0915):

15 09 * * * /bin/bash /home/username/Downloads/Spotify/spot_check.sh 2>&1

.bashrc:

if [ -f ~/.spotcheck ]; then
cd $HOME
./.spotcheck
fi

The bash shell – Linux terminal keyboard shortcuts

Hey!

Recently found out that there are some quite awesome shortcuts available that I’ve missed – extremely useful if you use the bash shell in *Nix.

To see which one you are using you can type ‘chsh’ in a terminal – this will tell you if you are using /bin/bash.

Go try them out! This can speed up your work in the terminal incredibly.

** Updated 2011-04-05 – there are many moooore!
See this little link: http://www.faqs.org/docs/bashman/bashref_93.html#SEC100

You can see the bindings by typing “bind -P”

The easier shortcuts:

Ctrl + A Go to the beginning of the line you are currently typing on. Extremely useful in those scenarios when left/right arrow keys do not work. Same as HOME button.
Ctrl + E Go to the end of the line you are currently typing on. Same as END button.
Ctrl + L Clears the Screen, same as the clear command.
Ctrl + U Clears the line before the cursor position. If you are at the end of the line, clears the entire line.
Ctrl + H Backspace.
Ctrl + R Let’s you search through previously used commands. Hit again to roll through the hits in the history. Searches through .bash_history in the user’s home directory.
Arrowkeys Up/Down Same as CTRL + P and CTRL + N. This will browse through the history. Hit enter to execute the command.
Ctrl + C Kill whatever you are running.
Ctrl + D Exit the current shell – logout.
Ctrl + Z Puts whatever you are running into a suspended background process. You can then use the terminal for something else.
Type ‘fg’ in the terminal to restore the process.
Ctrl + W Delete the word before the cursor.
Ctrl + K Clear the line after the cursor.
Ctrl + T Swap the last two characters before the cursor.
Esc + T Swap the last two words before the cursor.
Alt + F Move cursor forward one word on the current line.
Alt + B Move cursor backward one word on the current line.
Tab Auto-complete files and folder names.
Shift + Page Up / Down Scrolls through terminal buffer.

 

The shortcuts that are a little trickier:

Ctrl + X * In a directory you have two files: awesomeapp1.deb and notawesomebutneededapp4.deb. You want to install both. In debian the program you call is “dpkg -i filename.deb”. If you don’t want to write out all the names, you can type this: dpkg -i *.deb CTRL+x * (first ctrl+x and then press the * on your numpad or * on your normal, like shift+’) and then it will resolve the names so that your command will be “dkpg -i awesomeapp1.deb notawesomebutneededapp4.deb”
Ctrl + X Ctrl + V Prints something like this: GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
Macros CTRL +X ( to start, CTRL +X ) to save. For example: first hit CTRL+x SHIFT+8 (this is on my keyboard layout) – then put in your commands. Everything you type after this is saved in a macro. Then CTRL+X SHIFT+9 to save. Then hit CTRL+x e to run the macro.

Have fun!