Starbeamrainbowlabs

Stardust
Blog


Archive

Mailing List Articles Atom Feed Comments Atom Feed Twitter

Tag Cloud

3d account algorithms announcement archives arduino artificial intelligence assembly async audio bash batch blog bookmarklet booting c sharp c++ challenge chrome os code codepen coding conundrums coding conundrums evolved command line compiling css dailyprogrammer debugging demystification distributed computing downtime embedded systems encryption es6 features event experiment external first impressions future game github github gist graphics hardware meetup holiday html html5 html5 canvas interfaces internet io.js jabber javascript js bin labs learning library linux low level lua maintenance network networking node.js operating systems performance photos php pixelbot portable privacy programming problems project projects prolog protocol pseudo 3d python reddit reference release releases resource review rust secrets security series list server servers software sorting source code control svg technical terminal textures three thing game three.js tool tutorial twitter ubuntu university update upgrade version control visual web website windows windows 10 xmpp

How to update your linux kernel version on a KumSufi server

(Or why PHP throws random errors in the latest update)

Hello again!

Since I had a bit of a time trying to find some clear information on the subject, I'm writing the blog post so that it might help others. Basically, yesterday night I updated the packages on my server (the one that runs this website!). There was a PHP update, but I didn't think much of it.

This morning, I tried to access my ownCloud instance, only to discover that it was throwing random errors and refusing to load. I'm running PHP version 7.0.16-1+deb.sury.org~xenial+2. It was spewing errors like this one:

PHP message: PHP Fatal error:  Uncaught Exception: Could not gather sufficient random data in /srv/owncloud/lib/private/Security/SecureRandom.php:80
Stack trace:
#0 /srv/owncloud/lib/private/Security/SecureRandom.php(80): random_int(0, 63)
#1 /srv/owncloud/lib/private/AppFramework/Http/Request.php(484): OC\Security\SecureRandom->generate(20)
#2 /srv/owncloud/lib/private/Log/Owncloud.php(90): OC\AppFramework\Http\Request->getId()
#3 [internal function]: OC\Log\Owncloud::write('PHP', 'Uncaught Except...', 3)
#4 /srv/owncloud/lib/private/Log.php(298): call_user_func(Array, 'PHP', 'Uncaught Except...', 3)
#5 /srv/owncloud/lib/private/Log.php(156): OC\Log->log(3, 'Uncaught Except...', Array)
#6 /srv/owncloud/lib/private/Log/ErrorHandler.php(67): OC\Log->critical('Uncaught Except...', Array)
#7 [internal function]: OC\Log\ErrorHandler::onShutdown()
#8 {main}
  thrown in /srv/owncloud/lib/private/Security/SecureRandom.php on line 80" while reading response header from upstream, client: x.y.z.w, server: ownc

That's odd. After googling around a bit, I found this page on the Arch Linux bug tracker. I'm not using arch (Ubuntu 16.04.2 LTS actually), but it turned out that this comment shed some much-needed light on the problem.

Basically, PHP have changed the way they ask the Linux Kernel for random bytes. They now use the getrandom() kernel function instead of /dev/urandom as they did before. The trouble is that getrandom() was introduced in linux 3.17, and I was running OVH's custom 3.14.32-xxxx-grs-ipv6-64 kernel.

Thankfully, after a bit more digging, I found this article. It suggests installing the kernel you want and moving one of the grub config file generators to another directory, but I found that simply changing the permissions did the trick.

Basically, I did the following:


apt update
apt full-upgrade
apt install linux-image-generic
chmod -x /etc/grub.d/06_OVHkernel
update-grub
reboot

Basically, the above first updates everything on the system. Then it installs the linux-image-generic package. linux-image-generic is the pseudo-package that always depends on the latest stable kernel image available.

Next, I remove execute privileges on the file /etc/grub.d/06_OVHkernel. This is the file that gives the default installed OVH kernel priority over any other instalaled kernels, so it's important to exclude it from the grub configuration process.

Lastly, I update my grub configuration with update-grub and then reboot. You need to make sure that you update your grub configuration file, since if you don't it'll still use the old OVH kernel!

With that all done, I'm now running 4.4.0-62-generic according to uname -a. If follow these steps yourself, make sure you have a backup! While I am happy to try and help you out in the comments below, I'm not responsible for any consequences that may arise as a result of following this guide :-)

Fancy message of the day over SSH

Since my time to sit down for a good chunk of time and write some code has been extremely limited as of late, I've been playing around with a few smaller projects. One of those is a fancy message of the day when you log into a remote machine (in my case the server this website is hosted on!), and I thought I'd share it here.

My take on a fancy SSH message of the day.

The default message shown at the top when you login via ssh is actually generated by something called update-motd, and is generated from a set of scripts in /etc/update-motd.d. By customising these scripts, we can do almost anything we like!

To start off with, I disabled the execution of all the scripts in the directory (sudo chmod -x /etc/update-motd.d/*), and created a subfolder to store the script in that actually generated the system information (sudo mkdir /etc/update-motd.d/parts). Here's the script I wrote to generate the system information:

#!/usr/bin/env bash

. /etc/lsb-release

LOAD=$(cat /proc/loadavg | cut -d' ' -f 2);

CPU_COUNT=$(cat /proc/cpuinfo | grep -i "core id" | uniq | wc -l);
THREAD_COUNT=$(cat /proc/cpuinfo | grep -i "core id" | wc -l);

APT_UPDATE_DETAILS="$(/usr/lib/update-notifier/apt-check --human-readable | fold -w 40 -s)"

IPV4_ADDRESS=$(dig +short myip.opendns.com A @resolver1.opendns.com)
IPV6_ADDRESS=$(dig +short myip.opendns.com AAAA @2620:0:ccc::2);

LAST_LOGIN=$(last -1 | head -n 1 | awk '{ print $1,"at",$4,$5,$6,$7,"from",$3 }');

REBOOT_REQUIRED=$(/usr/lib/update-notifier/update-motd-reboot-required);

echo 
echo Welcome to $(hostname)
echo "  running ${DISTRIB_DESCRIPTION}"
echo 
echo Kernel: $(uname -r)
echo Uptime: $(uptime --pretty | sed -e 's/up //')
echo Load: ${LOAD}
echo 
echo IPs: ${IPV4_ADDRESS}, ${IPV6_ADDRESS}
echo 
echo "${APT_UPDATE_DETAILS}"
echo 
echo "${REBOOT_REQUIRED}"
#echo 
#echo Last login: ${LAST_LOGIN}

exit 0

Basically, I collect a bunch of information from random places on my system (several of which were taken from the existing scripts in /etc/update-motd.d/) and re-output them in a different format.

Then, I converted an image of my favicon logo with the brilliant catimg by posva to a set of unicode characters and sent that to a file (catimg -w 35 image.png >/etc/update-motd.d/sbrl-logo.txt) - you could alternatively use some ascii art from the internet (e.g. this site). Once done, I put the two together with the following script directly in my /etc/update-motd.d/ folder:

#!/usr/bin/env bash

### Settings ###
TMP_FILENAME=/run/sysinfo.txt

#/etc/update-motd.d/parts/sysinfo

################

/etc/update-motd.d/parts/sysinfo >$TMP_FILENAME

### Output ###
echo 
pr -mtJ /etc/update-motd.d/sbrl-logo.txt $TMP_FILENAME

##############

### Cleanup ###
rm $TMP_FILENAME

###############

Finally, I manually cleared and regenerated the message of of the day with sudo update-motd, giving the result you see at the top of this blog post. I also made sure to re-enable the execution of the other scripts I didn't use in my fancy motd so as to not miss out on their notifications.

If you're interested, I've generated an archive of my final /etc/update-motd.d folder (minus my logo in text format), which you can find here: 20170203-Fancy-Motd.7z.

Can you do better? Got a cool enhancement of your own? Post about it below!

Developing and Running C# Programs on Linux

Recently I was asked about running C# on Linux, and I remembered that I haven't actually written a blog post on it! This is that blog post I never wrote: A beginner's guide on how to develop, compile, and run C# programs on Linux. Here I assume a debian-based system (specifically Ubuntu 16.04), but it can be just as easily adapted to work with other flavours.

To start off, you'll need to add the mono repository to your system's apt repository list. Taken from the official mono website, here are the commands to do that:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb http://download.mono-project.com/repo/debian wheezy main" | sudo tee /etc/apt/sources.list.d/mono-xamarin.list
sudo apt update

The more experienced may notice that although I'm installing on a, Ubuntu 16.04 system, I'm still specifying wheezy when installing mono. This shouldn't make too much of a difference - currently mono only supports wheezy upon install.

Next, it's time to actually install mono itself. This is easy:

sudo apt install mono-mcs

The above ought to install the mono runtime and compiler. If you experience issues down the line, simply install the mono-complete package instead.With this installed, you should now be able to launch compiled programs written C♯ by double-clicking them in your file manager - without any modifications. C♯ programs compiled on Windows can be run on Linux, and vice versa (this is because C♯ actually compiled into something called the Common Intermediate Language, or CIL).

If you're on a server, then you can run your programs by prefixing the executable name with mono, like this:

mono program-of-awesomeness.exe

If your program crashes, however, the output is not very helpful at all. Thankfully though there's a way to remedy that. First, make sure your program is compiled in debug mode, and then add the --debug flag when running your program:

mono --debug another-awesome-program.exe

This is all very well, but what about compiling? That's relatively easy as well. mcs is the linux version of csc on Windows, and behaves almost identically with some minor syntactical changes (read up about it by typing mcs --help or man mcs). For Visual Studio solutions, there's xbuild. Xbuild is a new-ish build too for Linux that is capable of compiling almost any Visual Studio solution file without any modifications (though there are some undocumented difficulties that you might run into as an advanced user).

To use it, first install the mono-xbuild package (sudo apt install mono-xbuild), and then, in a terminal, cd into the directory that contains the solution you want to compile, and then type xbuild and hit enter. That's it!

All this work in the terminal is cool for running C♯ programs on GUI-less boxes and servers, but it's no way to develop a larger application. Thankfully, there's a solution to that too! Monodevelop is the best C♯ IDE out there at the moment - it's like Visual Studio for LInux. It's easy to install, too - simply install the monodevelop package (sudo apt install monodevelop).

Monodevelop.

(Above: Monodevelop running on my Linux laptop. The project open here is my sprite packing tool.)

The package in the official repositories should be good enough for general use (though it's probably out of date). For the latest version with all the latest features though, you'll have to compile it from source.

Sadly this is not a trivial process. To do it you need to be comfortable with the terminal and know your way reasonably well around a Linux system. If you still want to go ahead anyway, start by downloading the latest release and follow the instructions. You'll probably find it keeps complaining about things not existing - usually a quick apt search {thing} reveals which package you need to install in order to get it to work. If you have trouble, post a comment below and I'll try to help you out.

Even without compiling monodevelop from source, it's still a pretty good IDE. It lets you create Visual-Studio-compatible solution files, and compile your code on the fly at the touch of a button.

That just about covers the basics of running C♯ on Linux. If there's anything I've missed or you'd like to ask about, post a comment below!

Importing your friends' public keys automatically with gpg

Today's post is just a quick one, as I've had a rather busy week that I'm now recovering from :)

Several times over the past few weeks I've been finding myself googling around to try and find a way to download someone's GPG / PGP public key and import it into GPG automatically, so that I can verify their GPG signatures. I'm posting the command here so that I don't have to keep looking for it :D

Here's the command to search for a someone's public key:


gpg --keyserver pgp.mit.edu --search bill@billsboosters.com

The above asks the key server at pgp.mit.edu for GPG keys associated with the email address bill@billsboosters.org. You don't actually have to provide the keyserver - GPG will default to searching keys.gnupg.net.

Sources and further reading

Demystifying traceroute

A map of undersea cables. (Image from labnol. View the full map at submarinecablemap.com.)

A little while ago someone I know seemed a little bit confused as to how a traceroute interacts with a firewall, so I decided to properly look into and write this post.

Traceroute is the practice of sending particular packets across a network in order to discover all the different hops between the source computer and the destination. For example, here's a traceroute between starbeamrainbowlabs.com and bbc.co.uk:

traceroute to bbc.co.uk (212.58.244.23), 30 hops max, 60 byte packets
 1  125.ip-37-187-66.eu (37.187.66.125)  0.100 ms  0.015 ms  0.011 ms
 2  be10-147.rbx-g1-a9.fr.eu (37.187.231.169)  0.922 ms  0.912 ms  0.957 ms
 3  be100-1187.ldn-1-a9.uk.eu (91.121.128.87)  7.536 ms  7.538 ms  7.535 ms
 4  * * *
 5  ae-1-3104.ear2.London2.Level3.net (4.69.143.190)  18.481 ms  18.676 ms  18.903 ms
 6  unknown.Level3.net (212.187.139.230)  10.725 ms  10.434 ms  10.415 ms
 7  * * *
 8  ae0.er01.telhc.bbc.co.uk (132.185.254.109)  10.565 ms  10.666 ms  10.603 ms
 9  132.185.255.148 (132.185.255.148)  12.123 ms  11.781 ms  11.529 ms
10  212.58.244.23 (212.58.244.23)  10.596 ms  10.587 ms  65.243 ms

As you can see, there are quite a number of hops between us and the BBC, not all of which responded to attempts to probe them. Before we can speculate as to why, it's important to understand how a traceroute is performed.

There are actually a number of different methods to perform a traceroute, but they all have a few things in common. The basic idea exploits something called time to live (TTL). This is a special value that all IP packets have (located 16 bytes into an ipv4 header, and 7 bytes into an ipv6 header for those who are curious) that determines the maximum number of hops that a packet is allowed to go through before it is dropped. Every hop along a packet's route decreases this value by 1. When it reaches 0, an ICMP TTL Exceeded message is returned to the source of the packet. This message can be used to discover the hops between a given source and destination.

With that out of the way, we can move on to the different methods of generating this response from every hop along a given route. Linux comes with a traceroute utility built-in, and this is the tool that I'm going to be investigating. If you're on Windows, you can use tracert, but it doesn't have as many options as the Linux version.

Linux's traceroute utility defaults to using UDP packets on an uncommon port. It defaults to this because it's the best method that unprivileged users can use if they have a kernel older than 3.0 (check your kernel version with uname -r). It isn't ideal though, because many hosts don't expect incoming UDP packets and silently drop them.

Adding the -I flag causes traceroute to use ICMP ping requests instead. Thankfully most hosts will respond to ICMP pings, making it a much better probing tool. Some networks, however, don't allow ping requests to pass through their gateways (usually large institutions and schools), rendering this method useless in certain situations.

To combat the above, a new method was developed that uses TCP SYN packets instead of UDP or ICMP ping. If you send a TCP SYN packet (manipulating the TTL as above), practically all hosts will return some kind of message. This is commonly referred to as the TCP half-open technique, and defaults to port 80 - this allows the traceroute to bypass nearly all firewalls. If you're behind a proxy though I suspect it'll snag on it - theoretically speaking using port 443 instead should rectify this problem in most cases (i.e. traceroute -T -p 443 hostname.tld).

Traceroute has a bunch of other less reliable methods, which I'll explain quickly below.

  • -U causes traceroute to use UDP on port 53. This method usually only elicits responses from DNS servers along the route.
  • -UL makes traceroute use udplite in a similar fashion to UDP in the bullet point above. This is only available to administrators.
  • DCCP can also be used with the -D. It works similar to the TCP method described earlier.
  • A raw IP packet can also be used, but I can't think of any reasons you'd use this.

That just about covers all the different traceroute methods. If you have any questions, please leave a comment below.

3 Things to do before you upgrade to Ubuntu 16

Since Ubuntu 16.04 long term support has been released, I naturally have been wanting to upgrade to it. However, due to one thing or another, it hasn't been going so well. I decided to put this post together so that you don't make the same mistakes I did. Most of these tips I'm going to share with you in this post can be applied to be major upgrade process, although I found them useful when upgrading my Laptop to Ubuntu 16.04.

Make a backup. Seriously. I can't stress this enough. If the upgrade process gets interrupted or any reason, you'll be left with an unstable system - and a restore will be practically your only option.

Personally I use Timeshift to keep snapshots of my operating system files, and the default Déjà Dup backup program to backup my files. That way I can restore my OS to an earlier state without touching my files at all. Similarly, I can search though the history of my files without the clutter of a bunch of system files getting in the way.

Run the upgrader in either screen or tmux. Screen or Tmux let you run commands in a virtual window, and reattach to that window later if you loose your connection unexpectedly. In my case, I managed to lose access to the upgrader script halfway through the upgrade process, forcing me to kill the process and start over.

Make a checklist of things to do / test after the upgrade. Whilst upgrading, you'll find you start building up a long list of things you want to do and test after the upgrade is finished, such as reactivating extra PPAs, or merging config files. I recommend you write them all down somewhere so that you don't forget something.

That's about all I can think of at the moment. What upgrade are you in the process of doing? Do you have any tips to share? Write a comment below!

Resources

Recording animated gifs on Linux with Silentcast

Using Silentcast

A few months ago I was asked how I created animated gifs on Linux, and I said that I use silentcast. I also said that I'd write a blog post on it. I've been very busy since then, but now I have found some time when I remembered to post about it and am not exhausted.

Silentcast is a very versatile screen recording application that outputs either a set of png images, an animated gif, or 2 different types of video. It uses png files to store frames, so it isn't suitable for recording at a high fps or for very long, but it is still brilliant for recording short clips for your blog or to accopany a bug report.

Silentcast's dialogs stay in front of everything else that you have open, so you don't need to worry about loosing the window somewhere along the line. It integrates nicely with the Unity desktop (I haven't tried others yet), which makes it feel more intuitive and makes it easer to use. It also allows you to modify the intermediate png files before the final product is stitched together, too, allowing for precise edits to make the resulting gif loop perfectly.

It is written in bash, which makes it perfectly suited for usage on both Mac and Linux system , but I don't think that Windows is supported as of the time of posting. The other issue is that it took me a little while to work out how to record a custom area - this is done by the "Transparent Window Interior" option under "Area to be recorded". I also find it to be a little bit unpoliished around the edges (the icon especially needs some work), but overall it is an excellent piece of software that makes recording an animated gif on Linux a breeze - it's streets ahead of any other competing projects.

The animated gif above was taken and modified from Silentcast's GitHub project page.

Drive Naming Schemes in Linux

If you've used linux before, you'll probably have seen your flash drive or hard drive appearing as /dev/sdc4 at one point or another. You may have a cd drive which appeared as /dev/sr2 or /dev/cdrom. If you're really lucky, you might even have a tape drive that appears as /dev/st0.

What do all of these letters mean? I was wondering the same thing, so I looked it up and am writing up what I found in this post.

It would appear that in linux devices have a prefix and then an identifier. Before I explain the identifiers, I should outline each of the prefixes that I've come across (or looked up) first. You can find them below.

sd*

By far the most common device name prefix that I have come across is sd. This apparently originally stood for SCSI (SATA?) Device, but today it represents any regular block device that can be used for storage. This includes hard drives and flash drives for example.

hd*

The hd prefix originally stood for Hard Disk and was used for IDE drives, but it was dropped as of Linux 2.6.19 and sd* is now used instead.

sr*

sr stands for SCSI ROM and is used for CD / DVD drives (It might be used for blu-rays too, but I don't have one to check. Please post in the comments if you do!). On my machine, I have a number of symbolic links leading back to this drive in my /dev called cdrom, cdrw, dvd, and dvdrw.

st*

I have not come across a device with this prefix yet. If you have one, please post in the comments below! It stands for SCSI Tape drive.

Identifiers

The numbers and / or letters after the prefix refer to the device number, and, if appropriate, partition on the device. For example, /dev/sdb4 refers to the 4th partition on the 2nd disk, /dev/sr2 refers to the 3rd CD / DVD drive, and /dev/hdc3 refers to the 3rd partition on the 3rd IDE hard drive.

Sources

Adding to the $PATH when sudoing

Normally to add to your $PATH permenantly you do something like this: export PATH=$PATH:/new/dir. You may notice though tht this doesn't always seem to work. Here's an example:

$ export PATH=$PATH:/home/sbrl/bin
$ newcommand # works
$ sudo bash
# newcommand # works
# exit
$ sudo newcommand # doesn't work...wait what?

When a command is run with sudo, it doesn't respect your existing $PATH variable. To change it, you need to open the file /etc/sudoers and add your path to the line that says something like

Defaults    secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin
:/bin"

In order for your folder to be in your $PATH when you use sudo command.

UFW and Samba

Today I have another post for you about Samba. Today I found that people couldn't actually access the samba shares I set up (I must have forgotten to test them). They were getting a weird "The Network Path was not found" error. Strange. After looking into it, I found that I didn't unblock the right ports in ufw. You see, Samba operates using two listeners, one called smbd, and the another called nmbd. I had forgotten to read the output of netstat -peanut correctly, and I missed a few ports.

For future reference (and for others having the same problem), here's the list of commands you need to enter in order to use shared folders with Samba correctly:

sudo ufw allow 139/tcp
sudo ufw allow 445/tcp
sudo ufw allow 137/udp
sudo ufw allow 138/udp

Hopefully it doesn't take you as long to fix your problem as it did mine...!

Art by Mythdael