Starbeamrainbowlabs

Stardust
Blog


Archive

Mailing List Articles Atom Feed Comments Atom Feed Twitter

Tag Cloud

3d account algorithms announcement archives arduino artificial intelligence assembly async audio bash batch blog bookmarklet booting c sharp c++ challenge chrome os code codepen coding conundrums coding conundrums evolved command line compiling css dailyprogrammer debugging demystification distributed computing downtime embedded systems encryption es6 features event experiment external first impressions future game github github gist graphics hardware meetup holiday html html5 html5 canvas interfaces internet io.js jabber javascript js bin labs learning library linux low level lua maintenance network networking node.js operating systems performance photos php pixelbot portable privacy programming problems project projects prolog protocol pseudo 3d python reddit reference release releases resource review rust secrets security series list server servers software sorting source code control svg technical terminal textures three thing game three.js tool tutorial twitter ubuntu university update upgrade version control visual web website windows windows 10 xmpp

The HTTPS version of my website is insecure? Nonsense!

A chrome privacy warning.

I'm still rather ill, but I wanted to post about an issue I've just had with my website. Upon visting my website in the latest version of chrome beta (57 as of the time of typing), I discovered that chrome had decided that the connection was 'insecure'. It didn't tell me precisely what the problem was (even in the developer tools :-) - why would I possibly need to know that? - only that it considered it insecure.

After googling around a bit, I didn't find any specific articles on the subject - their recent move to start considering regular http connections insecure is swamping all the relevant articles in the search results I suspect.

The big clue came when I discovered that one of my subdomains that uses a letsencrypt works as expected. You see, the main website actually used a StartSSL certificate. My running theory is that even though my certificate was an SHA2 cerrtificate, chrome decided that it was not trustworthy as there was an SHA1 certificate in the trust chain somewhere.

The fix: Replace all my existing StartSSL certificates with Let's Encrypt ones. It seems to have fixed the issue for now. I also discovered that Let's Encrypt certificates can also be used in mail servers (i.e. SMTP and IMAP) too - so I don't have to go and fiddle about with finding an alternative certificate provider.

In future, it would certainly be helpful if Google actually told people precisely what they were going to do before they do it....!

Was this useful? Could it be improved? Would you like a Let's encrypt tutorial? Let me know in the comments below!

Learning Prolog Series List

The new Learning Prolog banner!

I'm rather ill at the moment, but I had the idea of creating a banner for the Learning Prolog series I did a while ago, and at the same time I ended up realising that I never posted a full series list for it. This banner has been something to occupy my mind with while I recover, at least.

Anyway, here's the series list:

What's a package manager? An introduction

A selection of package manager logos. From left to right: Bower, npm ruby gems, nuget, and pypi

If you haven't written much code outside of university yet, then you probably won't have heard of the concept, but today package mangers are everywhere. Every language and ecosystem has one. C♯ has Nuget, Javascript has npm and Bower, and Debian has apt. Even PHP now has one, called composer.

If you're thinking something along the lines of "Why do I need someone to take my parcels to the post office for me?", don't worry! I intend to explain what a package manager is and why / when you would want to use one in this post.

How many times have you downloaded a zip of someone else's library, and manually extracted it into your project? Perhaps it inflated the size of your version control repository, and slowed things down. This is just one of the problems that a package manager is designed to solve. Let me explain.

Imagine that you've just built a house. You've got all the walls and floors in place, and all that's left is to call the gas man and get some central heating installed. You call the gas man, and he comes out with his equipment and gets ready to install a brand new central heating system when he asks you about an electricity supply.

Oh, gosh! You totally forgot about that. Then you have to call the electrician and get him to come out and install the electricity supply. When he comes out, he asks whether you've got planning permission... and, well, you get the picture.

Imagine if you could just say "install this library!", and it would install it, handling all the dependencies for you. Imagine if you could store a list of libraries and their versions, such that you don't have to commit all your libraries to version control - saving on space and speeding up your repository. What if you could issue a single command, and all your libraries could be brought up to the latest version? Now, with a package manager, you can!

Most programming languages and operating systems you come across will have one, and it's usually a good idea to make use of it. To end this post, I'm going to list a few common languages, their associated package managers, and a link to a tutorial on how to use it.

  • C♯ - Nuget (Visual Studio tutorial)
  • Javascript - npm (tutorial, should work for Linux, Mac, and Windows users)
  • Ruby - RubyGems (tutorial, Linux users should have the gem command installed already along with Ruby)
  • Python - pip (tutorial - pip comes with python on both Linux and Windows. pypi - The Python Package Index. pip installs packages that are in the python package index.
  • Debian - apt (tutorial, should work for most Debian-based linux distributions - apt is probably the command behind your software manager gui. Not technically a programming language library package manager, but really useful all the same.)
  • PHP - composer (tutorial)

Need help? Want to say hello? Post below!

PixelBot Part 2: Devices need protocols, apparently

A selection of the technologies I'm using to put together my PixelHub server.

So there I was. I'd just got home, turned on my laptop, opened the Arduino IDE and Monodevelop, and then.... nothing. I knew I wanted my PixelBot to talk to the PixelHub I'd started writing, but I was confused as to how I could make it happen.

In this kind of situation, I realised that although I knew what I wanted them to do, I hadn't figured out the how. As it happens, when you're trying to get one (or more, in this case) different devices to talk to each other, there's something rather useful that helps them all to speak the same language: a protocol.

A protocol is a specification that defines the language that different devices use to talk to each other and exchange messages. Defining one before you start writing a networked program is probably a good idea - I find particularly helpful to write a specification for the protocol that the program(s) I'm writing, especially if their function(s) is/are complicated.

To this end, I've ended up spending a considerable amount of time drawing up the PixelHub Protocol - a specification document that defines how my PixelHub server is going to talk to a swarm of PixelBots. It might seem strange at first, but I decided on a (mostly) binary protocol.

Upon closer inspection though, (I hope) it makes a lot of sense. Since the Arduino is programmed using C++ as it has a limited amount of memory, it doesn't have any of the standard string manipulation function that you're used to in C♯. Since C++ is undoubtedly the harder of the 2 to write, I decided to make it easier to write the C++ rather than the C&sharp. Messages on the Arduino side are come in as a byte[] array, so (in theory) it should be easy to pick out certain known parts of the array and cast them into various different fundamental types.

With the specification written, the next step in my PixelBot journey is to actually implement it, which I'll be posting about in the next entry in this series!

How to update your linux kernel version on a KumSufi server

(Or why PHP throws random errors in the latest update)

Hello again!

Since I had a bit of a time trying to find some clear information on the subject, I'm writing the blog post so that it might help others. Basically, yesterday night I updated the packages on my server (the one that runs this website!). There was a PHP update, but I didn't think much of it.

This morning, I tried to access my ownCloud instance, only to discover that it was throwing random errors and refusing to load. I'm running PHP version 7.0.16-1+deb.sury.org~xenial+2. It was spewing errors like this one:

PHP message: PHP Fatal error:  Uncaught Exception: Could not gather sufficient random data in /srv/owncloud/lib/private/Security/SecureRandom.php:80
Stack trace:
#0 /srv/owncloud/lib/private/Security/SecureRandom.php(80): random_int(0, 63)
#1 /srv/owncloud/lib/private/AppFramework/Http/Request.php(484): OC\Security\SecureRandom->generate(20)
#2 /srv/owncloud/lib/private/Log/Owncloud.php(90): OC\AppFramework\Http\Request->getId()
#3 [internal function]: OC\Log\Owncloud::write('PHP', 'Uncaught Except...', 3)
#4 /srv/owncloud/lib/private/Log.php(298): call_user_func(Array, 'PHP', 'Uncaught Except...', 3)
#5 /srv/owncloud/lib/private/Log.php(156): OC\Log->log(3, 'Uncaught Except...', Array)
#6 /srv/owncloud/lib/private/Log/ErrorHandler.php(67): OC\Log->critical('Uncaught Except...', Array)
#7 [internal function]: OC\Log\ErrorHandler::onShutdown()
#8 {main}
  thrown in /srv/owncloud/lib/private/Security/SecureRandom.php on line 80" while reading response header from upstream, client: x.y.z.w, server: ownc

That's odd. After googling around a bit, I found this page on the Arch Linux bug tracker. I'm not using arch (Ubuntu 16.04.2 LTS actually), but it turned out that this comment shed some much-needed light on the problem.

Basically, PHP have changed the way they ask the Linux Kernel for random bytes. They now use the getrandom() kernel function instead of /dev/urandom as they did before. The trouble is that getrandom() was introduced in linux 3.17, and I was running OVH's custom 3.14.32-xxxx-grs-ipv6-64 kernel.

Thankfully, after a bit more digging, I found this article. It suggests installing the kernel you want and moving one of the grub config file generators to another directory, but I found that simply changing the permissions did the trick.

Basically, I did the following:


apt update
apt full-upgrade
apt install linux-image-generic
chmod -x /etc/grub.d/06_OVHkernel
update-grub
reboot

Basically, the above first updates everything on the system. Then it installs the linux-image-generic package. linux-image-generic is the pseudo-package that always depends on the latest stable kernel image available.

Next, I remove execute privileges on the file /etc/grub.d/06_OVHkernel. This is the file that gives the default installed OVH kernel priority over any other instalaled kernels, so it's important to exclude it from the grub configuration process.

Lastly, I update my grub configuration with update-grub and then reboot. You need to make sure that you update your grub configuration file, since if you don't it'll still use the old OVH kernel!

With that all done, I'm now running 4.4.0-62-generic according to uname -a. If follow these steps yourself, make sure you have a backup! While I am happy to try and help you out in the comments below, I'm not responsible for any consequences that may arise as a result of following this guide :-)

Coding Conundrums Evolved 5: Space Transmissions

A pretty procedurally generaated nebula. Code not written by me! (Banner from here with the seed 64oekm9bi4cg.)

This post is part of a biweekly series of programming problems, targeted at those learning to program using C# at University. The problems will vary in difficulty - some will be rather easy, and some will be quite challenging! Don't worry if you can't solve a problem just yet - come back to it in a few months and it'll seem a lot easier than it did before.

Coding Conundrums Evolved is back for another episode! I can't promise that these will be a regular feature on here due to the amount of time they take to put together, but I had a good idea for another one that I really couldn't turn down :D

The 2 new elligon-class starships, called the Lilai and the Inveinya, are on a joint mission deep in the newly-discovered Tenobulet Nebula. Due to the thickness of the gas clouds surrounding the 2 ships, they are having difficulty communicating, as talking over a video-link is causing the quantum neural-network based interference analysis system is draining all their available power clearing up the incoming data streams.

To combat this, the genius communications officer Mr. Ümafa has devised the Dassma System. The system allows the 2 ships to communicate by sending text-based messages, reducing power usage by sending a quantum coaxial error correction stream in parallel to the actual message stream.

Anyway, Mr. Ümafa needs your help as the newly-hired systems programming engineer, since all the other programmers onboard are busy scratching their heads over a new asymmetric inverse shift register algorithm whilst writing a firmware upgrade for the quadrilateral energy transmission stabilisers to increase their efficiency.

Unfortunately, his dassma system can only transmit messages in short chunks of 128 characters, and he's finding it terribly cumbersome to keep splitting up the long messages that the crew keep asking him to send.

Given a long string as input, write a program that splits the input into multiple chunks that are at most 128 characters long, and outputs each chunk on a separate line.

Helpful Hints

  • Perform every step of the process on paper first.
  • Draw a flowchart of what your program needs to do.
  • If you're having trouble, step through your code line by line and inspect it's state at each point.

Challenge

Mr. Ümafa is impressed with your program, but he's got an additional request or two. Unfortunately, Communications Officer Bramaar of the Iveinya keeps mixing up the different message parts, and he finds it annoying when it splits a message in the middle of a word.

Mr. Ümafa would like you to upgrade your program so that it only splits on spaces, and so that it labels each of the chunks with their number in the sequence.

The solution archive is here. I'll release the password to this challenge in the comments in 2 weeks time.

Writing code when you don't have the time

As you've probably noticed, posts around here have slowed down recently. There's a reason: I've been very busy doing a year in industry. Currently, my goal is to release one post a week. While my time has been rather fragmented and at times extremely limited, I've still been able to sit down for a little while here and there to write some blog posts and some code (If I can actually pull it off, I've got a seriously cool project I'm going to post about on here in the near-ish future!).

Due in part to the fact that I really don't want to exclusively write code at my industrial placement, I've been trying my hardest to keep programming and playing around with things in my free time. It's not as easy as you might think. Sometimes, the setup and teardown time eats all the time I allocated away so can't actually get anything done.

If this sounds a little bit like your time at the moment, fear not! I have developed a technique or two I wanted to share on here, just in case someone else finds it useful :-)

Planning what it is that you want to do is really important. You probably know this already, but it is especially so if you don't have a ton of time to throw at a project, because otherwise you can easily spend longer figuring out what you need to do next than actually doing it. I try to break my projects down into small, manageable bite-sized chunks that I can tackle one at a time. Only have 1/2 an hour at a time? Break it down into portions that will take you about 1/2 an hour complete. It might take a while, but breaking your project down can help it go a little bit faster.

Even with breaking my project down, I often find myself forgetting where I got to last time. To tackle this, I've discovered that leaving a comment in the file I was last editing explaining in a sentence or two what I need to do next helps me figure it out faster. It's also really useful that my editor (whichever one I'm using at the time) is configured to remember the files I had open last - letting me quickly pick up where I left off. Monodevelop, Visual Studio, and Atom do this automatically - if your editor doesn't, there's bound to be a setting or an extension that does it for you.

By planning what I need to do next, and leaving myself short comments explaining what I was about to do next, I can increase the amount of time I spend actually writing code instead of fumbling around working out what I wanted to do next. It's certainly not an ideal way to program, but with practice you can get quite proficient at it....

Found this helpful? Got any tips yourself? Comment down below!

Fancy message of the day over SSH

Since my time to sit down for a good chunk of time and write some code has been extremely limited as of late, I've been playing around with a few smaller projects. One of those is a fancy message of the day when you log into a remote machine (in my case the server this website is hosted on!), and I thought I'd share it here.

My take on a fancy SSH message of the day.

The default message shown at the top when you login via ssh is actually generated by something called update-motd, and is generated from a set of scripts in /etc/update-motd.d. By customising these scripts, we can do almost anything we like!

To start off with, I disabled the execution of all the scripts in the directory (sudo chmod -x /etc/update-motd.d/*), and created a subfolder to store the script in that actually generated the system information (sudo mkdir /etc/update-motd.d/parts). Here's the script I wrote to generate the system information:

#!/usr/bin/env bash

. /etc/lsb-release

LOAD=$(cat /proc/loadavg | cut -d' ' -f 2);

CPU_COUNT=$(cat /proc/cpuinfo | grep -i "core id" | uniq | wc -l);
THREAD_COUNT=$(cat /proc/cpuinfo | grep -i "core id" | wc -l);

APT_UPDATE_DETAILS="$(/usr/lib/update-notifier/apt-check --human-readable | fold -w 40 -s)"

IPV4_ADDRESS=$(dig +short myip.opendns.com A @resolver1.opendns.com)
IPV6_ADDRESS=$(dig +short myip.opendns.com AAAA @2620:0:ccc::2);

LAST_LOGIN=$(last -1 | head -n 1 | awk '{ print $1,"at",$4,$5,$6,$7,"from",$3 }');

REBOOT_REQUIRED=$(/usr/lib/update-notifier/update-motd-reboot-required);

echo 
echo Welcome to $(hostname)
echo "  running ${DISTRIB_DESCRIPTION}"
echo 
echo Kernel: $(uname -r)
echo Uptime: $(uptime --pretty | sed -e 's/up //')
echo Load: ${LOAD}
echo 
echo IPs: ${IPV4_ADDRESS}, ${IPV6_ADDRESS}
echo 
echo "${APT_UPDATE_DETAILS}"
echo 
echo "${REBOOT_REQUIRED}"
#echo 
#echo Last login: ${LAST_LOGIN}

exit 0

Basically, I collect a bunch of information from random places on my system (several of which were taken from the existing scripts in /etc/update-motd.d/) and re-output them in a different format.

Then, I converted an image of my favicon logo with the brilliant catimg by posva to a set of unicode characters and sent that to a file (catimg -w 35 image.png >/etc/update-motd.d/sbrl-logo.txt) - you could alternatively use some ascii art from the internet (e.g. this site). Once done, I put the two together with the following script directly in my /etc/update-motd.d/ folder:

#!/usr/bin/env bash

### Settings ###
TMP_FILENAME=/run/sysinfo.txt

#/etc/update-motd.d/parts/sysinfo

################

/etc/update-motd.d/parts/sysinfo >$TMP_FILENAME

### Output ###
echo 
pr -mtJ /etc/update-motd.d/sbrl-logo.txt $TMP_FILENAME

##############

### Cleanup ###
rm $TMP_FILENAME

###############

Finally, I manually cleared and regenerated the message of of the day with sudo update-motd, giving the result you see at the top of this blog post. I also made sure to re-enable the execution of the other scripts I didn't use in my fancy motd so as to not miss out on their notifications.

If you're interested, I've generated an archive of my final /etc/update-motd.d folder (minus my logo in text format), which you can find here: 20170203-Fancy-Motd.7z.

Can you do better? Got a cool enhancement of your own? Post about it below!

Easier TCP Networking in C♯

I see all sorts of C♯ networking tutorials out there telling you that you have to use byte arrays and buffers and all sorts of other complicated things if you ever want to talk to another machine over the network. Frankly, it's all rather confusing.

Thankfully though, it doesn't have to stay this way. I've learnt a different way of doing TCP networking in C♯ at University (thanks Brian!), and I realised the other day I've never actually written a blog post about it (that I can remember, anyway!). If you know how to read and write files and understand some basic networking concepts (IP addresses, ports, what TCP and UDP are, etc.), you'll have no problems understanding this.

Server

The easiest way to explain it is to demonstrate. Let's build a quick server / client program where the server says hello to the client. Here's the server code:


// Server.cs
using System;
using System.Net;
using System.Net.Sockets;
using System.Threading.Tasks;
using System.IO;

public class Server
{
    public readonly int Port;
    public Server(int inPort)
    {
        Port = inPort; string s;
    }

    public async Task Start()
    {
        TcpListener server = new TcpListener(IPAddress.Any, Port);
        server.Start();
        while (true) {
            TcpClient nextClient = await server.AcceptTcpClientAsync();
            StreamReader incoming = new StreamReader(nextClient.GetStream());
            StreamWriter outgoing = new StreamWriter(nextClient.GetStream()) { AutoFlush = true };

            string name = (await incoming.ReadLineAsync()).Trim();
            await outgoing.WriteLineAsync($"Hello, {name}!");

            Console.WriteLine("Said hello to {0}", name);

            nextClient.Close();
        }
    }
}

// Use it like this in your Main() method:
Server server = new Server(6666);
server.Start().Wait();

Technically speaking, that asynchronous code ought to be running in a separate thread - I've omitted it to make it slightly simpler :-) Let's break this down. The important bit is in the Start() method - the rest is just sugar around it to make it run if you want to copy and paste it. First, we create & start a TcpListener:

TcpListener server = new TcpListener(IPAddress.Any, Port);
server.Start();

Once done, we enter a loop, and wait for the next client:

TcpClient nextClient = await server.AcceptTcpClientAsync();

Now that we have a client to talk to, we attach a StreamReader and a StreamWriter with a special option set on it to allow us to talk to the remote client with ease. The option set on the StreamWriter is AutoFlush, and it basically tells it to flush it's internal buffer every time we write to it - that way things we write to it always hit the TcpClient underneath. Depending on your setup the TcpClient does some internal buffering & optimisations anyway, so we don't need the second layer of buffering here:

StreamReader incoming = new StreamReader(nextClient.GetStream());
StreamWriter outgoing = new StreamWriter(nextClient.GetStream()) { AutoFlush = true };

With that, the rest should be fairly simple to understand:

string name = (await incoming.ReadLineAsync()).Trim();
await outgoing.WriteLineAsync($"Hello, {name}!");

Console.WriteLine("Said hello to {0}", name);

nextClient.Close();

First, we grab the first line that the client sends us, and trim of any whitespace that's lurking around. Then, we send back a friendly hello message to client, before logging what we've done to the console and closing the connection.

Client

Now that you've seen the server code, the client code should be fairly self explanatory. The important lines are highlighted:


using System;
using System.Net;
using System.Net.Sockets;
using System.Threading.Tasks;
using System.IO;

public class Client
{
    public readonly string Hostname;
    public readonly int Port;

    public Client(string inHostname, int inPort)
    {
        Hostname = inHostname; uint a;
        Port = inPort;
    }

    public async Task GetHello(string name) {
        TcpClient client = new TcpClient();
        client.Connect(Hostname, Port);

        StreamReader incoming = new StreamReader(client.GetStream());
        StreamWriter outgoing = new StreamWriter(client.GetStream()) { AutoFlush = true };

        await outgoing.WriteLineAsync(name);

        return (await incoming.ReadLineAsync()).Trim();
    }
}

// Use it like this in your Main() method:
Client client = new Client("localhost", 6666);
Console.Write("Enter your name: ");
Console.WriteLine("The server said: {0}", client.GetHello(Console.ReadLine()).Result);

First, we create a new client and connect it to the server. Next, we connect the StreamReader and StreamWriter instances to the TcpClient, and then we send the name to the server. Finally, we read the response the server sent us and return it. Easy!

Here's some example outputs:

Client:


./NetworkingDemo-Server.exe
Said hello to Bill

Server:


./NetworkingDemo-Client.exe
Enter your name: Bill
The server said: Hello, Bill!

The above code should work on Mac, Windows, and Linux. Granted, it's not the most efficient way of doing things, but it should be fine for most general purposes. Personally, I think the trade-off between performance and readability/ease of understanding of code is totally worth it.

If you prefer, I've got an archive of the above code I wrote for this blog post - complete with binaries. You can find it here: NetworkingDemo.7z.

Coding Conundrums Evolved Series List

A combination of all the banner images form the series so far. Left to right: ep 1, 2, 3, and then finally 4. Since a while ago I didn't have the time to write the next episode of Coding Conundrums Evolved and then I kind of forgot about it (sorry! I really haven't had much time since September this year, and they take ages to write... :-(), I've decided post a series list for the 4 episodes I've posted so far.

Before I forget though, the (long overdue!) password for last episode's solution (#4) is illykin. Remember to attempt the challenge yourself first before looking at my solution! Working out how you'd attempt a problem is just as important as actually writing the solution itself.

With the last password revealed, here's the series list:

While I'm certainly not averse to releasing a new entry in this series, it'll probably have to wait until the end of this year - unless I think of a really good idea that can't wait. Suggestions and ideas are welcome though :-)

Art by Mythdael