Achievement Get: Upgrade Server - A writeup of my experiences

The upgrade is complete! I've managed to move practically everything over to the new server, apart from a few automated cleanup tasks here and there. There are still a few issues floating around, but they shouldn't affect this website, and I should have them cleared up soon. Although the migration went smoother than I expected, I did encounter some issues and learnt a few things that I thought I'd share here.

The first thing I found was that starting a todo list isn't a rather good idea. It sounds simple, but it's actually really useful. I found that I had a lot of small tasks I needed to complete, and I kept thinking of more things that needed doing at regular intervals. If I didn't write them down I'd never get anything done because I'd never be able to remember what needed doing first!

It also helps to do your research before you move. Make sure that you're properly reaquainted with all the software running on the system that you're going to migrate from, and that you're familiar with all the ins and outs of your particular situation. If you aren't, then you risk stumbling across some rather nasty, complicated, and time consuming problems mid-migration.

It also helps to do as much of the migration as possible without taking the old system offline. Install the software, Move the configuration files. Set up the firewall. Set up your new monitoring tools. This allows you to minimise the downtime that you have to subject your users to, which is always a good thing.

Lastly, testing is incredibly important. test everything. Make sure that every little feature after you migrate. You'd be surprised at how many issues can crop up after migration.

Server migration!

I've been watching Kimsufi's server page for what feels like absolutely ages now, waiting to get my hands on an ultra-cheap €4.99 per month (excluding VAT of course) KS-1 dedicated server. Unfortunately I've never been quite fast enough, so yesterday I decided that enough was enough and went ahead and bought a KS-2B at €9.99 per month (again excluding VAT). After all is said and done it works out to about £8.39 per month, which, for 2 cores / 4 threads, 4GB RAM, and a 40GB SSD, is an absolute bargain in my eyes.

I've been busy moving things across and it's going well, but I haven't finished yet - I still have the web server and the mail server to set up. I'm also looking at using the Hiawatha webserver instead of Nginx. Hiawatha is a security-first and easy to configure web server. Apparently it's also lightweight, but we'll see about that...! Nginx's config files have been annoying me for a while now, so I think that it's high time I tried something different.

Use C♯ 6.0 today in Visual Studio 2013

(Banner from hdw.eweb4.com)

In case you haven't heard, C♯ 6.0 is here now and it's awesome (here's a cheat sheet from programmingwithmosh.com showing the most noteable new features). Unfortunately, you must be using either Visual Studio 2015 or above or MonoDevelop in order to take advantage of it.... until now: Microsoft have released their C♯ 6.0 compiler Roslyn as a NuGet package.

If you don't know what a NuGet package is, Nuget is a modular system that allows you to pull in and use various different libraries and tools automatically. There's a central registry over at nuget.org, which people (like you!) can upload their packages to and other people can download them from. This looks like a good tutorial for Windows users. MonoDevelop users need to install this addin, but it should be installed already.

All you have to do is install the Microsoft.Net.Compilers NuGet package in order to use C♯ 6.0 in Visual Studio 2013. That's it! Unfortunately, this breaks the build process on platforms other than Windows, as the MicroSoft.Net.Compilers package is Windows only. The solution is fairly simple however. Once you've installed the above NuGet package, open your ".csproj" file in your favourite plain text editor (such as Notepad or gedit), and find the line that looks like this (it should be near the bottom):

<Import Project="..\packages\Microsoft.Net.Compilers.1.3.2\build\Microsoft.Net.Compilers.props" Condition="Exists('..\packages\Microsoft.Net.Compilers.1.3.2\build\Microsoft.Net.Compilers.props')" />

And add AND '$(OS)' == 'Windows_NT' to the end of the Condition attribute like this:

  <Import Project="..\packages\Microsoft.Net.Compilers.1.3.2\build\Microsoft.Net.Compilers.props" Condition="Exists('..\packages\Microsoft.Net.Compilers.1.3.2\build\Microsoft.Net.Compilers.props') AND '$(OS)' == 'Windows_NT'" />

The above adds a condition that prevents the compiler in NuGet package you installed from being used on platforms other than Windows. This doesn't mean that you can't use C♯ 6.0 on other platforms - Mono (the Linux c♯ compiler) already supports C♯ 6.0 natively, so it doesn't need to be replaced. it's just the C♯ compiler bundled with Visual Studio 2013 and below that's no good.

Set properties faster in C♯

Recently I rediscovered that C♯ lets you set multiple properties when you create an instance of an object. I've been finding it useful, so I thought that I'd share it here.

Consider this code:

StreamWriter outgoingData = new StreamWriter("rockets.txt");
outgoingData.AutoFlush = true;
outgoingData.Encoding = Encoding.UTF8;

In the above I create a new StreamWriter and attach it to the file rockets.txt. I also turn on AutoFlush, and set the encoding to UTF8. The code above is starting to look a little messy, so let's rewrite it:

StreamWriter outgoingData = new StreamWriter("rockets.txt") {
    AutoFlush = true,
    Encoding = Encoding.UTF8
};

This technique can come in particularly handy when you need to set a lot of properties on an object upon it's creation.You can also do it with other types, too:

string[] animals = new string[] { "cat", "mouse", "elephant" };
List<int> primes = new List<int>() { 2, 3, 5, 7, 11, 13, 17, 19 };
Dictionary<string, IPAddress> ips = new Dictionary<string, IPAddress>() {
    { "google.com", Dns.GetHostAddresses("google.com")[0] }
    { "wikipedia.org", Dns.GetHostAddresses("wikipedia.org")[0] }
    { "starbeamrainbowlabs.com", Dns.GetHostAddresses("starbeamrainbowlabs.com")[0] }
    { "hull.ac.uk", Dns.GetHostAddresses("hull.ac.uk")[0] }
}

Further Reading

• Object and Collection Initializers from the MSDN
• How to: Initialize a Dictionary with a Collection Initializer from the MSDN

Set and forget async tasks

(Banner image from here by GDJ)

Recently I've been using asynchronous C# quite a bit, and I've run into the problem of 'setting and forgetting' an asynchronous task more than once. You might want to do this when handling requests in some sort of server, for example.

I looked into it and came up with a few snippets of code I thought someone else might find useful, so I'm posting them here.

Without further delay, here's the first snippet:

/// <summary>
/// Call this method to allow a given task to complete in the background.
/// Errors will be handled correctly.
/// Useful in fire-and-forget scenarios, like a TCP server for example.
/// From http://stackoverflow.com/a/22864616/1460422
/// Adapted by Starbeamrainbowlabs
/// </summary>
/// <param name="task">The task to forget about.</param>
/// <param name="acceptableExceptions">Acceptable exceptions. Exceptions specified here won't cause a crash.</param>
public static async void ForgetTask(Task task, params Type[] acceptableExceptions)
{
    try
    {
        await task.ConfigureAwait(false);
    }
    catch (Exception ex)
    {
        // TODO: consider whether derived types are also acceptable.
        if (!acceptableExceptions.Contains(ex.GetType()))
            throw;
    }
}

All asynchronous methods in C♯ return some form of Task - and these Task s can be reconfigured and manipulated to make them run in the background on the thread pool, as in the above. The above also handles exceptions correctly so that your asynchronous methods won't just silently fail.

Talking about exceptions, if you await an asynchronous method, it's highly likely that if they do throw an exception it'll be an AggregateException. This is not helpful. It doesn't tell us anything about the actual exception that was thrown in the first place! It gets annoying manually inspecting the innerExceptions property of the AggregateException very quickly. Thankfully, I've found a solution to that too:

try
{
    await DoAsyncWork();
}
catch(AggregateException agError)
{
    agError.Handle((error) => {
        ExceptionDispatchInfo.Capture(error).Throw();
        throw error;
    });
}
catch
{
    Console.Error.WriteLine("Something went very wrong O.o");
    throw;
}

I can't remember where I found the ExceptionDispatchInfo bit (if it was your idea, please let me know so I can give you appropriate credit!), but the rest I wrote myself. It essentially unwraps the AggregateException and rethrows each exception in turn, whilst preserving the original stack trace. That way you can track the issue that threw the exception in the first place down.

Coding Conundrums Series List

I was just looking through my blog's archive, and I found a series of posts I made that solve Rob Miles' Coding Condundrums that he posted on his Python Site. Since I seem to have forgotten to post a series list (it's likely that I finished the series before I thought of such a thing!), I thought that it's better late than never, so here's a series list of all the problems I've solved:

• Coding Conundrums 1
• Coding Conundrums 2
• Coding Conundrums 3 (Announcement post)

If all goes to plan, I might start a new (possibly related!) series around the end of September.

I built a robot!

The day before yesterday we had another hardware meetup at C4DI, and I built a robot! Now I just have to figure out how to program it...

It's one of Rob Miles' Hull Pixel Bots, with a Wemos D1 R2 on the top and two stepper motors and their driver boards mounted in the middle, and a battery box on the bottom.

The wheels are a little wonky, but we'll sort that out next time :) For now, I'm going to have some fun making it run around on the floor :D

The web needs encrypting, but that's not the whole story

There's been a lot of talk about 'encrypting the web' recently, so I decided to post about it myself. In this post I'll explain the problem, the solution, and why the solution may cause more problems than it solves.

The problem itself has actually been around since the web was designed, but it hasn't really been a huge problem until recently. Let's consider a legitimate origin server that serves HTML files about animals, such as bunnies.html. Since it doesn't have anything to hide (who'd want to steal pictures of bunnies, anyway?), it serves these files over regular HTTP, not HTTPS.

Suppose that our animal website becomes hugely popular overnight, and has thousands of users browsing it per hour. Let's also suppose that someone has managed to gain control of the network that connects our origin server's network to the wider internet. Since this someone is smart, rather than sending a DOS attack or a ton of spam from the compromised network, rather they fiddle about with the responses that our origin server is sending to, turning our unsuspecting users' browsers into the source of a massive DDOS attack or worse! A prime example of this was the attack on GitHub last year.

If a website's communications aren't encrypted, it allows anyone to inspect and tamper with both requests and responses. You can't even guarantee that you're connecting to the server you think you are and not a cleverly designed imitation!

HTTPS solves all of these problems. It encrypts communications between the client and server, preventing messages from being inspected or tampered with. It also verifies the identity of the server, which is why you need a certificate in order to serve things over HTTPS.

The other side

At the beginning of this post, I said that HTTPS isn't the whole story. Obviously it solves the problem at hand, right? Yes it does, but it also brings to other issues to the table: complicating the setup process, and breaking links. The second of these problems is easy - we can just setup automatic redirects that send you to the HTTPS version of a site.

The first problem is decidedly more difficult to solve. HTTPS is difficult to set up, and if it becomes the default, it could reduce the accessibility of setting up and running your own website. Thankfully it isn't the required yet.

At the moment, nobody has a complete solution to this issue. Letsencrypt is a new brilliant service that makes obtaining SSL certificates easy, provided that you can either fiddle with your web server config, or are willing to let a script do it for you, but it doesn't help you set it up correctly (another excellent read).

The other slightly related issue is that users often mistake 'HTTPS' to mean secure. While this is true of the communications between their computer and the server, it doesn't stop the server from setting the root password to 1234 and storing passwords in plain text.

If you're still here, thank you for reading! Hopefully you now understand some of the issues surrounding web security at the moment. Please post a comment down below if you have anything to say (-:

Sources and further reading

• https://thenanfang.com/why-baidu-was-hijacked-to-attack-github/
• An introduction to javascript based DDOS by Cloudflare
• I used Ascidia by Frimkron to render the diagram above.

Importing your friends' public keys automatically with gpg

Today's post is just a quick one, as I've had a rather busy week that I'm now recovering from :)

Several times over the past few weeks I've been finding myself googling around to try and find a way to download someone's GPG / PGP public key and import it into GPG automatically, so that I can verify their GPG signatures. I'm posting the command here so that I don't have to keep looking for it :D

Here's the command to search for a someone's public key:

gpg --keyserver pgp.mit.edu --search bill@billsboosters.com

The above asks the key server at pgp.mit.edu for GPG keys associated with the email address bill@billsboosters.org. You don't actually have to provide the keyserver - GPG will default to searching keys.gnupg.net.

Sources and further reading

• Using GPG with Evolution by the Fedora Project
• Where to upload PGP public key? Are KeyServers still surviving? asked by RPK on superuser
• seahorse-nautilus - Nautilus integration for GPG (Direct install link)

Embedding files in C# binaries

You've probably got a whole load of files that your latest program needs to run. You could set them to 'Copy to output directory' and then read them in from there, but this not only causes clutter in your program's output directory, but also potentially exposes files to your users that you don't want them fiddling with or deleting.

The solution: embed them in the program itself! It sounds complex, but it's actually simpler than you'd imagine. In this post I'll show you how to embed files in your solutions and how to interact it them at runtime.

Firstly, you need to set the build action for any files that you'd like to embed to be 'Embedded Resource'. This is slightly different in MonoDevelop and Visual Studio.

In Visual Studio, single click the file in the solution explorer, go to the properties window, find 'Build Action', and select 'Embedded Resource'.

In MonoDevelop, right click the file in the solution explorer, go to the 'Build Action', and then select 'EmbeddedResource'.

If you need to edit the .csproj file directly, add the following to the bottom of the <project> tag:

<ItemGroup>
  <EmbeddedResource Include="relative/path/to/filename.ext" />
  <EmbeddedResource Include="relative/path/to/another/filename.ext2" />
</ItemGroup>

It supports wildcards with an asterisk (*) too, if you want to add a whole folder automatically.

Once done, the file will be embedded directly into the parent project's binary when you next hit build. This isn't much use without being able to read it in though - let's do that next.

When files are embedded in C♯ a prefix gets added to them based on the default namespace of the project they are a part of and the folder that they are in. The easiest way to discover what this prefix has been set to is to list all of the files that have been embedded:

string[] resNames = Assembly.GetExecutingAssembly().GetManifestResourceNames();
foreach (string resName in resNames)
    Console.WriteLine(resName);

If the above looks a little unfamiliar to you, don't worry. It uses reflection to get a list of files embedded in the currently executing assembly (you'll need to add using System.Reflection; to the top of your file in order to use it too). Reflection is a brilliant tool to obtain all sorts of information about the execution environment you are running in. It's worth a look if you haven't checked it out already.

Adjust the above accordingly if you don't have a console in your project. Now that we've got the name of our embedded file, with some work we can attach a StreamReader to it and read it like any other file:

public static string GetEmbeddedResourceContent(string resourceName)
{
    Assembly asm = Assembly.GetExecutingAssembly();
    Stream stream = asm.GetManifestResourceStream(resourceName);
    StreamReader source = new StreamReader(stream);
    string fileContent = source.ReadToEnd();
    source.Dispose();
    stream.Dispose();
    return fileContent;
}

In a similar fashion to the previous code sample, in the above we get a reference to the currently executing binary, and then we ask it for the Stream that's attached to the embedded file that we want to read. After that, we can attach a StreamReader to it and read the whole thing all at once.

That's all you need to do to embed files into your programs and read them back in again. Sadly you can't write to embedded resources at runtime (they're read only - it would require self-modification), so if you need to change them you could instead unpack them from the binary on load and them modify the unpacked version instead.

If there's anything that you'd like to see me cover, please let me know in the comments below.