Archive

## Tag Cloud

3d account algorithms announcement archives arduino artificial intelligence assembly async audio bash batch blog bookmarklet booting c sharp c++ challenge chrome os code codepen coding conundrums coding conundrums evolved command line compiling css dailyprogrammer debugging demystification distributed computing downtime embedded systems encryption es6 features event experiment external first impressions future game github github gist graphics hardware hardware meetup holiday html html5 html5 canvas interfaces internet io.js jabber javascript js bin labs learning library linux low level lua maintenance network networking node.js operating systems performance photos php pixelbot portable privacy programming problems project projects prolog protocol protocols pseudo 3d python reddit reference release releases resource review rust secrets security series list server software sorting source code control statistics svg technical terminal textures three thing game three.js tool tutorial tutorials twitter ubuntu university update updates upgrade version control virtualisation visual web website windows windows 10 xmpp

## /r/dailyprogrammer hard challenge #322: Static HTTP 1.0 server

Recently I happened to stumble across the dailyprogrammer subreddit's latest challenge. It was for a static HTTP 1.0 server, and while I built something similar for my networking ACW, I thought I'd give this one a go to create an extendable http server that I can use in other projects. If you want to follow along, you can find the challenge here!

My language of choice, as you might have guessed, was C♯ (I know that C♯ has a HttpServer class inbuilt already, but to listen on 0.0.0.0 on Windows it requires administrative privileges).

It ended up going rather well, actually. In a little less than 24 hours after reading the post, I had myself a working solution, and I thought I'd share here how I built it. Let's start with a class diagram:

(Above: A class diagram for the GlidingSquirrel. Is this diagram better than the last one I drew?)

I'm only showing properties on here, as I'll be showing you the methods attached to each class later. It's a pretty simple design, actually - HttpServer deals with all the core HTTP and networking logic, FileHttpServer handles the file system calls (and can be swapped out for your own class), and HttpRequest, HttpResponse, HttpMethod, HttpResponseCode all store the data parsed out from the raw request coming in, and the data we're about to send back out again.

With a general idea as to how it's put together, lets dive into how it actually works. HttpServer would probably be a good place to start:

public abstract class HttpServer
{
public static readonly string Version = "0.1-alpha";

public string BindEndpoint { /* ... */ }

protected TcpListener server;

private Mime mimeLookup = new Mime();
public Dictionary<string, string> MimeTypeOverrides = new Dictionary<string, string>() {
[".html"] = "text/html"
};

{ /* ... */ }
public HttpServer(int inPort) : this(IPAddress.IPv6Any, inPort)
{
}

public async Task Start() { /* ... */ }

public string LookupMimeType(string filePath) { /* ... */ }

protected async void HandleClientThreadRoot(object transferredClient) { /* ... */ }

public async Task HandleClient(TcpClient client) { /* ... */ }

public abstract Task HandleRequest(HttpRequest request, HttpResponse response);
}

(Full version)

It's heavily abbreviated because there's actually quite a bit of code to get through here, but you get the general idea. The start method is the main loop that accepts the TcpClients, and calls HandleClientThreadRoot for each client it accepts. I decided to use the inbuilt ThreadPool class to do the threading for me here:

TcpClient nextClient = await server.AcceptTcpClientAsync();
ThreadPool.QueueUserWorkItem(new WaitCallback(HandleClientThreadRoot), nextClient);

C♯ handles all the thread spawning and killing for me internally this way, which is rather nice. Next, HandleClientThreadRoot sets up a net to catch any errors that are thrown by the next stage (as we're now in a new thread, which can make debugging a nightmare otherwise), and then calls the main HandleClient:

try
{
await HandleClient(client);
}
catch(Exception error)
{
Console.WriteLine(error);
}
finally
{
client.Close();
}

No matter what happens, the client's connection will always get closed. HandleClient is where the magic start to happen. It attaches a StreamReader and a StreamWriter to the client:

StreamReader source = new StreamReader(client.GetStream());
StreamWriter destination = new StreamWriter(client.GetStream()) { AutoFlush = true };

...and calls a static method on HttpRequest to read in and decode the request:

HttpRequest request = await HttpRequest.FromStream(source);
request.ClientAddress = client.Client.RemoteEndPoint as IPEndPoint;

More on that later. With the request decoded, HandleClient hands off the request to the abstract method HandleRequest - but not before setting up a secondary safety net first:

try
{
await HandleRequest(request, response);
}
catch(Exception error)
{
response.ResponseCode = new HttpResponseCode(503, "Server Error Occurred");
await response.SetBody(
$"An error ocurred whilst serving your request to '{request.Url}'. Details:\n\n" +$"{error.ToString()}"
);
}

This secondary safety net means that we can send a meaningful error message back to the requesting client in the case that the abstract request handler throws an exception for some reason. In the future, I'll probably make this customisable - after all, you don't always want to let the client know exactly what crashed inside the server's internals!

The FileHttpServer class that handles the file system logic is quite simple, actually. The magic is in it's implementation of the abstract HandleRequest method that the HttpServer itself exposes:

public override async Task HandleRequest(HttpRequest request, HttpResponse response)
{
if(request.Url.Contains(".."))
{
await response.SetBody("Error the requested path contains dangerous characters.");
return;
}

string filePath = getFilePathFromRequestUrl(request.Url);
if(!File.Exists(filePath))
{
response.ResponseCode = HttpResponseCode.NotFound;
await response.SetBody($"Error: The file path '{request.Url}' could not be found.\n"); return; } FileInfo requestFileStat = null; try { requestFileStat = new FileInfo(filePath); } catch(UnauthorizedAccessException error) { response.ResponseCode = HttpResponseCode.Forbidden; await response.SetBody( "Unfortunately, the server was unable to access the file requested.\n" + "Details:\n\n" + error.ToString() + "\n" ); return; } response.Headers.Add("content-type", LookupMimeType(filePath)); response.Headers.Add("content-length", requestFileStat.Length.ToString()); if(request.Method == HttpMethod.GET) { response.Body = new StreamReader(filePath); } } With all the helper methods and properties on HttpResponse, it's much shorter than it would otherwise be! Let's go through it step by step. if(request.Url.Contains("..")) This first step is a quick check for anything obvious that could be used against the server to break out of the web root. There are probably other dangerous things you can do(or try to do, anyway!) to a web server to attempt to trick it into returning arbitrary files, but I can't think of any of the top of my head that aren't covered further down. If you can, let me know in the comments! string filePath = getFilePathFromRequestUrl(request.Url); Next, we translate the raw path received in the request into a path to a file on disk. Let's take a look inside that method: protected string getFilePathFromRequestUrl(string requestUrl) { return$"{WebRoot}{requestUrl}";
}

It's rather simplistic, I know. I can't help but feel that there's something I missed here.... Let me know if you can think of anything. (If you're interested about the dollar syntax there - it's called an interpolated string, and is new in C♯ 6! Fancy name, I know. Check it out!)

if(!File.Exists(filePath))
{
response.ResponseCode = HttpResponseCode.NotFound;
await response.SetBody($"Error: The file path '{request.Url}' could not be found.\n"); return; } Another obvious check. Can't have the server crashing every time it runs into a 404! A somewhat interesting note here: File.Exists only checks to see if there's a file that exists under the specified path. To check for the existence of a directory, you have to use Directory.Exists - which would make directory listing rather easy to implement. I might actually try that later - with an option to turn it off, of course. FileInfo requestFileStat = null; try { requestFileStat = new FileInfo(filePath); } catch(UnauthorizedAccessException error) { response.ResponseCode = HttpResponseCode.Forbidden; await response.SetBody( "Unfortunately, the server was unable to access the file requested.\n" + "Details:\n\n" + error.ToString() + "\n" ); return; } Ok, on to something that might be a bit more unfamiliar. The FileInfo class can be used to get, unsurprisingly, information about a file. You can get all sorts of statistics about a file or directory with it, such as the last modified time, whether it's read-only from the perspective of the current user, etc. We're only interested in the size of the file though for the next few lines: response.Headers.Add("content-type", LookupMimeType(filePath)); response.Headers.Add("content-length", requestFileStat.Length.ToString()); These headers are important, as you might expect. Browsers to tend to like to know the type of content they are receiving - and especially it's size. if(request.Method == HttpMethod.GET) { response.Body = new StreamReader(filePath); } Lastly, we send the file's contents back to the user in the response - but only if it's a GET request. This rather neatly takes care of HEAD requests - but might cause issues elsewhere. I'll probably end up changing it if it does become an issue. Anyway, now that we've covered everything right up to sending the response back to the client, let's end our tour with a look at the request parsing system. It's a bit backwards, but it does seem to work in an odd sort of way! It all starts in HttpRequest.FromStream. public static async Task<HttpRequest> FromStream(StreamReader source) { HttpRequest request = new HttpRequest(); // Parse the first line string firstLine = await source.ReadLineAsync(); var firstLineData = ParseFirstLine(firstLine); request.HttpVersion = firstLineData.httpVersion; request.Method = firstLineData.requestMethod; request.Url = firstLineData.requestPath; // Extract the headers List<string> rawHeaders = new List<string>(); string nextLine; while((nextLine = source.ReadLine()).Length > 0) rawHeaders.Add(nextLine); request.Headers = ParseHeaders(rawHeaders); // Store the source stream as the request body now that we've extracts the headers request.Body = source; return request; } It looks deceptively simple at first glance. To start with, I read in the first line, extract everything useful from it, and attach them to a new request object. Then, I read in all the headers I can find, parse those too, and attach them to the request object we're building. Finally, I attach the StreamReader to the request itself, as it's now pointing at the body of the request from the user. I haven't actually tested this, as I don't actually use it anywhere just yet, but it's a nice reminder just in case I do end up needing it :-) Now, let's take a look at the cream on the cake - the method that parses the first line of the incoming request. I'm quite pleased with this actually, as it's my first time using a brand new feature of C♯: public static (float httpVersion, HttpMethod requestMethod, string requestPath) ParseFirstLine(string firstLine) { List<string> lineParts = new List<string>(firstLine.Split(' ')); float httpVersion = float.Parse(lineParts.Last().Split('/')[1]); HttpMethod httpMethod = MethodFromString(lineParts.First()); lineParts.RemoveAt(0); lineParts.RemoveAt(lineParts.Count - 1); string requestUrl = lineParts.Aggregate((string one, string two) =>$"{one} {two}");

return (
httpVersion,
httpMethod,
requestUrl
);
}

Monodevelop, my C♯ IDE, appears to go absolutely nuts over this with red squiggly lines everywhere, but it still compiles just fine :D

As I was writing this, a thought popped into my head that a tuple would be perfect here. After reading somewhere a month or two ago about a new tuple syntax that's coming to C♯ I thought I'd get awesomely distracted and take a look before continuing, and what I found was really cool. In C♯ 7 (the latest and quite possibly greatest version of C♯ to come yet!), there's a new feature called value tuples, which let's you dynamically declare tuples like I have above. They're already fully supported by the C♯ compiler, so you can use them today! Just try to ignore your editor if it gets as confused as mine did... :P

If you're interested in learning more about them, I'll leave a few links at the bottom of this post. Anyway, back to the GlidingSquirrel! Other than the new value tuples in the above, there's not much going on, actually. A few linq calls take care of the heavy lifting quite nicely.

And finally, here's my header parsing method.

public static Dictionary<string, string> ParseHeaders(List<string> rawHeaders)
{
Dictionary<string, string> result = new Dictionary<string, string>();

{
KeyValuePair<string, string> nextHeader = new KeyValuePair<string, string>(
parts[0].Trim().ToLower(),
parts[1].Trim()
);
result[nextHeader.Key] = $"{result[nextHeader.Key]},{nextHeader.Value}"; else result[nextHeader.Key] = nextHeader.Value; } return result; } While I have attempted to build in support for multiple definitions of the same header according to the spec, I haven't actually encountered a time when it's actually been needed. Again, this is one of those things I've built in now for later - as I do intend on updating this and adding more features later - and perhaps even work it into another secret project I might post about soon. Lastly, I'll leave you with a link to the repository I'm storing the code for the GlidingSquirrel, and a few links for your enjoyment: GlidingSquirrel ### Sources and Further Reading ## Access your home linux box from anywhere with SSH tunnels (Header by GDJ from openclipart.org. Source page) ....and other things! Recently, I bought a Raspberry Pi 3. Now that the rest of the components have arrived, I've got a rather nice little home server that's got a 1 terabyte WD PiDrive attached to it to provide lots of lovely shared storage, which is rather nice. However, within a few weeks I was faced with a problem. How do I access my new box to configure it from my internship when I'm on lunch? Faced with such a challenge, I did what anyone would, and took to the internet to find a solution. It didn't take long. A while ago I heard about these things called 'SSH tunnels', which, while not designed for a high throughput, are more than adequate for a low-intensity SSH connection that runs a few kilobytes a second in either direction. After reading this excellent answer by erik on the Unix & Linux StackExchange, I had an understanding of how SSH tunnels work, and was ready to put together a solution. You should go and read that answer if you'd like to understand SSH tunnels too - it explains it much better than I ever could :P With that knowledge in hand, I went about planning the SSH tunnel. I already have a server a public IP address (it's hosting this website!), so I needed a reverse tunnel to allow me to access a port local to my linux box at home (called elessar - a virtual cookie for anyone who gets the reference!) from starbeamrainbowlabs.com. Important! Ask yourself whether it's moral and ethical to set up an ssh tunnel before you think about following along with this article! If you find yourself behind a firewall or something similar, then the chances are that it's there for a good reason - and you might get into trouble if you try and circumvent it. I won't be held responsible for any loss or damages of any description caused by the reading of this post. First job: create a limited account on starbeamrainbowlabs.com for elessar to SSH into. That's easy: sudo useradd --system ssh-tunnel Then, with a few quick lines in /etc/ssh/sshd_config: Match User ssh-tunnel ForceCommand echo 'This account can only be used for ssh tunnelling.' ....we can prevent the ssh-tunnel user from being abused to gain shell access to the server (let me know if there are any further measures I can put in place here). Now that I had a user account to ssh in as, I could set up a public / private keypair to authenticate with starbeamrainbowlabs.com, and cook up an SSH command for elessar that would set up the appropriate tunnel. After fiddling around a bit, I came up with this that did the job: ssh -TN -R30582:localhost:5724 ssh-tunnel@starbeamrainbowlabs.com Very cool. So with that command executing on elessar, I could ssh into elessar from starbeamrainbowlabs.com! In short, it sets up a tunnel that will make port 30582 on starbeamrainbowlabs.com tunnel through to port 5724 on elessar - the port on elessar that has SSH running on it, without allocating a pseudo-tty to save resources. explainshell.com can, well, explain it in more detail if you're interested. Having an SSH command that would set up the tunnel is nice, but it's not very useful, since I have to execute it first before I can actually SSH into elessar from afar. The solution was actually a little bit complicated. First, I wrote a simple systemd service file (systemd is what I have installed, since it's vanilla raspbian - this should be easily adaptable to other systems and setups) to start the SSH tunnel automagically on boot: [Unit] Description=SSH tunnel from starbeamrainbowlabs.com to local ssh server. [Service] Type=simple ExecStart=/usr/bin/ssh -TN -R30582:localhost:5724 ssh-tunnel@starbeamrainbowlabs.com [Install] WantedBy=network-online.target I quickly realised that there were a few flaws with this approach. Firstly, it tried to start the SSH connection before my router had connected to the internet, since my router starts faster than the box that initialises the fibre connection to my ISP. Secondly, it fails to retry when the connection dies. The first problem can be solved relatively easily, by wrapping the ssh command in a clever bit of shell scripting: /bin/sh -c 'until ping -c1 starbeamrainbowlabs.com &>/dev/null && sleep 5; do :; done && /usr/bin/ssh -TN -R30582:localhost:5724 ssh-tunnel@starbeamrainbowlabs.com The above tries to ping starbeamrainbowlabs.com every 5 seconds until it succeeds, and only then does it attempt to open the SSH connection. This solves the first problem. To solve the second, we need to look at autossh. Autossh is a small tool that monitors an ssh connection in a variety of configurable ways and restarts the connection if ever dies for whatever reason. You can install it with your favourite package manager: sudo apt install autossh Substitute apt with whatever package manager you use on your system. With it installed, we can use a command like this: autossh -o "UserKnownHostsFile /home/ssh-tunnel/.ssh/known_hosts" -o "IdentityFile /home/ssh-tunnel/.ssh/ssh-tunnel_ed25519" -o "PubkeyAuthentication=yes" -o "PasswordAuthentication=no" -o "ServerAliveInterval 900" -TN -R30582:localhost:5724 -p 7261 ssh-tunnel@starbeamrainbowlabs.com to automatically start our ssh tunnel, and restart it if anything goes wrong. Note all the extra settings I had to specify here. This is because even though I had many of them specified in ~/.ssh/config for the ssh-tunnel user, because of systemd's weird environment when it starts a service, I found I had to specify everything in the command line with absolute paths (ugh). Basically, the above tells autossh where the known_hosts file is (important for automation!), that it should only attempt public / private keypair authentication and not password authentication, that it should check the server's still there every 15 minutes, and all the other things we figured out above. Finally, I combined the solutions I came up with for both problems, which left me with this: [Unit] Description=SSH tunnel from starbeamrainbowlabs.com to local ssh server. [Service] Type=simple ExecStart=/bin/sh -c 'until ping -c1 starbeamrainbowlabs.com &>/dev/null && sleep 5; do :; done && /usr/bin/autossh -o "UserKnownHostsFile /home/pi/.ssh/known_hosts" -o "IdentityFile /home/pi/.ssh/ssh-tunnel_ed25519" -o "PubkeyAuthentication=yes" -o "PasswordAuthentication=no" -o "ServerAliveInterval 900" -TN -R30582:localhost:5724 -p 7261 ssh-tunnel@starbeamrainbowlabs.com' [Install] WantedBy=network-online.target Here's a version that utilises the -f parameter of autossh to put the autossh into the background, which eliminates the sh parent process: [Unit] Description=SSH tunnel from starbeamrainbowlabs.com to local ssh server. [Service] Type=forking Environment=AUTOSSH_PIDFILE=/var/run/sbrl-ssh-tunnel/ssh-tunnel.pid PIDFile=/var/run/sbrl-ssh-tunnel/ssh-tunnel.pid ExecStartPre=/bin/mkdir -p /var/run/sbrl-ssh-tunnel ExecStartPre=-/bin/chown ssh-tunnel:ssh-tunnel /var/run/sbrl-ssh-tunnel ExecStart=/bin/sh -c 'until ping -c1 starbeamrainbowlabs.com &>/dev/null && sleep 5; do :; done && /usr/bin/autossh -f -o "UserKnownHostsFile /home/pi/.ssh/known_hosts" -o "IdentityFile /home/pi/.ssh/ssh-tunnel_ed25519" -o "PubkeyAuthentication=yes" -o "PasswordAuthentication=no" -o "ServerAliveInterval 900" -TN -R30582:localhost:5724 -p 7261 ssh-tunnel@starbeamrainbowlabs.com' [Install] WantedBy=network-online.target I ended up further modifying the above to set up an additional tunnel to allow elessar to send emails via the postfix email server that's running on starbeamrainbowlabs.com. Let me know if you'd be interested in a tutorial on this! ### Sources and Futher Reading ## PixelBot Part 2: Devices need protocols, apparently So there I was. I'd just got home, turned on my laptop, opened the Arduino IDE and Monodevelop, and then.... nothing. I knew I wanted my PixelBot to talk to the PixelHub I'd started writing, but I was confused as to how I could make it happen. In this kind of situation, I realised that although I knew what I wanted them to do, I hadn't figured out the how. As it happens, when you're trying to get one (or more, in this case) different devices to talk to each other, there's something rather useful that helps them all to speak the same language: a protocol. A protocol is a specification that defines the language that different devices use to talk to each other and exchange messages. Defining one before you start writing a networked program is probably a good idea - I find particularly helpful to write a specification for the protocol that the program(s) I'm writing, especially if their function(s) is/are complicated. To this end, I've ended up spending a considerable amount of time drawing up the PixelHub Protocol - a specification document that defines how my PixelHub server is going to talk to a swarm of PixelBots. It might seem strange at first, but I decided on a (mostly) binary protocol. Upon closer inspection though, (I hope) it makes a lot of sense. Since the Arduino is programmed using C++ as it has a limited amount of memory, it doesn't have any of the standard string manipulation function that you're used to in C♯. Since C++ is undoubtedly the harder of the 2 to write, I decided to make it easier to write the C++ rather than the C&sharp. Messages on the Arduino side are come in as a byte[] array, so (in theory) it should be easy to pick out certain known parts of the array and cast them into various different fundamental types. With the specification written, the next step in my PixelBot journey is to actually implement it, which I'll be posting about in the next entry in this series! ## Easier TCP Networking in C♯ I see all sorts of C♯ networking tutorials out there telling you that you have to use byte arrays and buffers and all sorts of other complicated things if you ever want to talk to another machine over the network. Frankly, it's all rather confusing. Thankfully though, it doesn't have to stay this way. I've learnt a different way of doing TCP networking in C♯ at University (thanks Brian!), and I realised the other day I've never actually written a blog post about it (that I can remember, anyway!). If you know how to read and write files and understand some basic networking concepts (IP addresses, ports, what TCP and UDP are, etc.), you'll have no problems understanding this. ### Server The easiest way to explain it is to demonstrate. Let's build a quick server / client program where the server says hello to the client. Here's the server code:  // Server.cs using System; using System.Net; using System.Net.Sockets; using System.Threading.Tasks; using System.IO; public class Server { public readonly int Port; public Server(int inPort) { Port = inPort; string s; } public async Task Start() { TcpListener server = new TcpListener(IPAddress.Any, Port); server.Start(); while (true) { TcpClient nextClient = await server.AcceptTcpClientAsync(); StreamReader incoming = new StreamReader(nextClient.GetStream()); StreamWriter outgoing = new StreamWriter(nextClient.GetStream()) { AutoFlush = true }; string name = (await incoming.ReadLineAsync()).Trim(); await outgoing.WriteLineAsync($"Hello, {name}!");

Console.WriteLine("Said hello to {0}", name);

nextClient.Close();
}
}
}

// Use it like this in your Main() method:
Server server = new Server(6666);
server.Start().Wait();


Technically speaking, that asynchronous code ought to be running in a separate thread - I've omitted it to make it slightly simpler :-) Let's break this down. The important bit is in the Start() method - the rest is just sugar around it to make it run if you want to copy and paste it. First, we create & start a TcpListener:

TcpListener server = new TcpListener(IPAddress.Any, Port);
server.Start();

Once done, we enter a loop, and wait for the next client:

TcpClient nextClient = await server.AcceptTcpClientAsync();

Now that we have a client to talk to, we attach a StreamReader and a StreamWriter with a special option set on it to allow us to talk to the remote client with ease. The option set on the StreamWriter is AutoFlush, and it basically tells it to flush it's internal buffer every time we write to it - that way things we write to it always hit the TcpClient underneath. Depending on your setup the TcpClient does some internal buffering & optimisations anyway, so we don't need the second layer of buffering here:

StreamReader incoming = new StreamReader(nextClient.GetStream());
StreamWriter outgoing = new StreamWriter(nextClient.GetStream()) { AutoFlush = true };

With that, the rest should be fairly simple to understand:

string name = (await incoming.ReadLineAsync()).Trim();
await outgoing.WriteLineAsync($"Hello, {name}!"); Console.WriteLine("Said hello to {0}", name); nextClient.Close(); First, we grab the first line that the client sends us, and trim of any whitespace that's lurking around. Then, we send back a friendly hello message to client, before logging what we've done to the console and closing the connection. ### Client Now that you've seen the server code, the client code should be fairly self explanatory. The important lines are highlighted:  using System; using System.Net; using System.Net.Sockets; using System.Threading.Tasks; using System.IO; public class Client { public readonly string Hostname; public readonly int Port; public Client(string inHostname, int inPort) { Hostname = inHostname; uint a; Port = inPort; } public async Task GetHello(string name) { TcpClient client = new TcpClient(); client.Connect(Hostname, Port); StreamReader incoming = new StreamReader(client.GetStream()); StreamWriter outgoing = new StreamWriter(client.GetStream()) { AutoFlush = true }; await outgoing.WriteLineAsync(name); return (await incoming.ReadLineAsync()).Trim(); } } // Use it like this in your Main() method: Client client = new Client("localhost", 6666); Console.Write("Enter your name: "); Console.WriteLine("The server said: {0}", client.GetHello(Console.ReadLine()).Result);  First, we create a new client and connect it to the server. Next, we connect the StreamReader and StreamWriter instances to the TcpClient, and then we send the name to the server. Finally, we read the response the server sent us and return it. Easy! Here's some example outputs: Client:  ./NetworkingDemo-Server.exe Said hello to Bill  Server:  ./NetworkingDemo-Client.exe Enter your name: Bill The server said: Hello, Bill!  The above code should work on Mac, Windows, and Linux. Granted, it's not the most efficient way of doing things, but it should be fine for most general purposes. Personally, I think the trade-off between performance and readability/ease of understanding of code is totally worth it. If you prefer, I've got an archive of the above code I wrote for this blog post - complete with binaries. You can find it here: NetworkingDemo.7z. ## Picking the right interface for multicast communications At the recent hardware meetup, I was faced with an interesting problem: I was trying to communicate with my Wemos over multicast UDP in order to get it to automatically discover my PixelHub Server, but the multicast pings were going out over the wrong interface - I had both an ethernet cable plugged in and a WiFi hotspot running on my integrated wireless card. The solution to this is not as simple you might think - you have to not only pick the right interface, but also the right version of the IP protocol. You also have to have some way to picking the correct interface in the first place. Let's say you have a big beefy PC with a wireless card and 2 ethernet ports that are (for some magical reason) all in use at the same time, and you want to communicate with another device over your wireless card and not either of your ethernet ports. I developed this on my linux laptop, but it should work just fine on other OSes. To start, it's probably a good idea to list all of our network interfaces: using System.Net.NetworkInformation; // ... NetworkInterface[] nics = NetworkInterface.GetAllNetworkInterfaces(); foreach (NetworkInterface nic in nics) { Console.WriteLine("Id: {0} - Description: {1}", nic.Id, nic.Description); } This (on my machine at least!) outputs something like this: eth0 - eth0 lo - lo wlan0 - wlan0 Your machine will probably output something different. Next, since you can't normally address this list of network interfaces directly by name, we need to write a method to do it for us: public static NetworkInterface GetNetworkIndexByName4(string targetInterfaceName) { NetworkInterface[] nics = NetworkInterface.GetAllNetworkInterfaces(); foreach (NetworkInterface nic in nics) { if (nic.Id == targetInterfaceName) return nic; } throw new Exception($"Error: Can't find network interface with the name {targetInterfaceName}.");
}

Pretty simple, right? We're not out the woods yet though - next we need to tell our UdpClient to talk on a specific network interface. Speaking of which, let's set up that UdpClient so that we can use it to do stuff with multicast:

using System.Net;

// ...

UdpClient client = new UdpClient(5050);
client.JoinMulticastGroup(IPAddress.Parse("239.62.148.30"));

With that out of the way, we can now deal with telling the UcpClient which network interface it should be talking on. This is actually quite tricky, since the UdpClient doesn't take a NetworkInterface directly. Let's define another helper method:

public static int GetIPv4Index(this NetworkInterface nic)
{
IPInterfaceProperties ipProps = nic.GetIPProperties();
IPv4InterfaceProperties ip4Props = ipProps.GetIPv4Properties();
return ip4Props.Index;
}

The above extension method gets the index of the IPv4 interface of the network interface. Since at the moment we are in the middle of a (frustratingly slow) transition from IPv4 and IPv6, each network interface must have both an IPv4 interface, for talking to other IPv4 hosts, and an IPv6 interface for talking to IPv6 hosts. In this example I'm using IPv4, since the Wemos I want to talk to doesn't support IPv6 :-(

Now that we have a way to get the index of a network interface, we need to translate it into something that the UdpClient understands:

int interfaceIndex = (int)IPAddress.HostToNetworkOrder(NetTools.GetNetworkIndexByName4(NetworkInterfaceName));

That complicated! Thankfully, we don't need to pick it apart completely - it just works :-)

Now that we have the interface index in the right format, all we have to do is tell the UdpClient about it. Again, this is also slightly overcomplicated:

beacon.Client.SetSocketOption(
SocketOptionLevel.IP,
SocketOptionName.MulticastInterface,
interfaceIndex
);

Make sure that you put this call before you join the multicast group. With that done, your UdpClient should finally be talking on the right interface!

Whew! That was quite the rabbit hole (I sent my regards to the rabbit :P). If you have any issues with getting it to work, I'm happy to help - just post a comment down below.

### Sources

• Openclipart images: 1, 2

## Creating a HTTP Server in C♯

(Image from here.)

I discovered the HttpListener class in .NET the other day, and decided to take a look. It turns out that it's way easier to use than doing it all by hand as I had to do in my 08241 Networking module a few months ago. I thought I'd write a post about it so that you can see how easy it is and use the same technique yourself. You might want to add a HTTP status reporting system to your program, or use it to serve a front-end GUI. You could even build a real-time app with WebSockets.

As with all my tutorials - this post is a starting point, not an ending point. Refactor it to your heart's content!

To start off, let's create a new C♯ console project and create an instance of it:

using System;
using System.Net;

class MainClass
{
public static void Main(string[] args)
{
HttpListener listener = new HttpListener();
}
}

This should look pretty familiar to you. If not, then I can recommend the C♯ yellow book by Rob Miles. This is all rather boring so far, so let's move on quickly.

listener.Prefixes.Add("http://*:3333/");
listener.Start();

C♯'s HttpListener works on prefixes. It parses the prefix to work out what it should listen on and for what. The above listens for HTTP requests from anyone on port 3333. Apparently you can listen for HTTPS requests using this class too, but I'd recommend putting it behind some kind of proxy like NginX instead and let that handle the TLS certificates instead.

We also start the listener listening for requests too, because it doesn't listen for requests by default.


while(true)
{
HttpListenerContext cycle = listener.GetContext();
Console.WriteLine("Got request for {0} from {1}.", cycle.Request.RawUrl, cycle.Request.RemoteEndPoint);
StreamWriter outgoing = new StreamWriter(cycle.Response.OutputStream);
}


This next part is the beginning of the main request / response loop. The highlighted line is the important one - it retrieves the next pending request, waiting for it if necessary. The following lines log to the console about the latest request, and set up a StreamWriter to make it easier to send the response to the requester.

The final piece of the puzzle actually sending the request and moving onto the next one.

outgoing.WriteLine("It works!");
outgoing.Close();
cycle.Response.Close();

Actually writing the response is easy - all you have to do is write it to the StreamWriter. Then all you have to do is call Close() on the stream writer and the Response object, and you're done! Here's the whole thing:

using System;
using System.Net;
using System.IO;

namespace HttpServerTest
{
class MainClass
{
public static void Main(string[] args)
{
HttpListener listener = new HttpListener();
listener.Start();

while(true)
{
HttpListenerContext cycle = listener.GetContext();
Console.WriteLine("Got request for {0} from {1}.", cycle.Request.RawUrl, cycle.Request.RemoteEndPoint);
StreamWriter outgoing = new StreamWriter(cycle.Response.OutputStream);
cycle.Response.ContentType = "text/plain";
outgoing.WriteLine("It works!");
outgoing.Close();
cycle.Response.Close();
}
}
}
}


(Hastebin, Raw)

That's all you really need to create a simple HTTP server in C♯. You should extend this by putting everything that could go wrong in a try..catch statement to prevent the server from crashing. You could also write a simple find / replace templating system to make it easier to serve files from your new HTTP server. Refactoring the above to make it asynchronous wouldn't be a bad idea either.

Have you been building anything interesting recently? Post about in the comments below!

## TeleConsole: A simple remote debugging solution for awkward situations

Several times in the last few months I've found myself in some kind of awkward situation where I need to debug a C♯ program, but the program in question either doesn't have a console, or is on a remote machine. In an ideal world, I'd like to have all my debugging message sent to my development machine for inspection. That way I don't have to check the console of multiple different machines just to get an idea as to what has gone wrong.

C♯ already has System.Diagnostics.Debug, which functions similarly to the Console class, except that it sends data to the Application output window. This is brilliant for things running on your local machine through Visual Studio or MonoDevelop, but not so great when you've got a program that utilises the network and has to run on separate computers. Visual Studio for one starts to freeze up if you open the exact same repository on a network drive that's already open somewhere else.

It is for these reasons that I finally decided to sit down and write TeleConsole. It's a simple remote console that you can use in any project you like. It's split into 2 parts: the client library and the server binary. The server runs on your development machine, and listens for connections from the client library, which you reference in your latest and greatest project and use to send messages to the server.

Take a look here: sbrl/TeleConsole (GitHub) (Direct link to the latest release)

The client API is fully documented with intellisense comments, so I assume it should be very easy to work out how to use it (if there's something I need to do to let you use the intellisense comments when writing your own programs, let me know!). If you need some code to look at, I've created an example whilst testing it.

Although it's certainly not done yet, I'll be definitely be building upon it in the future to improve it and add additional features.

## Demystifying traceroute

(Image from labnol. View the full map at submarinecablemap.com.)

A little while ago someone I know seemed a little bit confused as to how a traceroute interacts with a firewall, so I decided to properly look into and write this post.

Traceroute is the practice of sending particular packets across a network in order to discover all the different hops between the source computer and the destination. For example, here's a traceroute between starbeamrainbowlabs.com and bbc.co.uk:

traceroute to bbc.co.uk (212.58.244.23), 30 hops max, 60 byte packets
1  125.ip-37-187-66.eu (37.187.66.125)  0.100 ms  0.015 ms  0.011 ms
2  be10-147.rbx-g1-a9.fr.eu (37.187.231.169)  0.922 ms  0.912 ms  0.957 ms
3  be100-1187.ldn-1-a9.uk.eu (91.121.128.87)  7.536 ms  7.538 ms  7.535 ms
4  * * *
5  ae-1-3104.ear2.London2.Level3.net (4.69.143.190)  18.481 ms  18.676 ms  18.903 ms
6  unknown.Level3.net (212.187.139.230)  10.725 ms  10.434 ms  10.415 ms
7  * * *
8  ae0.er01.telhc.bbc.co.uk (132.185.254.109)  10.565 ms  10.666 ms  10.603 ms
9  132.185.255.148 (132.185.255.148)  12.123 ms  11.781 ms  11.529 ms
10  212.58.244.23 (212.58.244.23)  10.596 ms  10.587 ms  65.243 ms

As you can see, there are quite a number of hops between us and the BBC, not all of which responded to attempts to probe them. Before we can speculate as to why, it's important to understand how a traceroute is performed.

There are actually a number of different methods to perform a traceroute, but they all have a few things in common. The basic idea exploits something called time to live (TTL). This is a special value that all IP packets have (located 16 bytes into an ipv4 header, and 7 bytes into an ipv6 header for those who are curious) that determines the maximum number of hops that a packet is allowed to go through before it is dropped. Every hop along a packet's route decreases this value by 1. When it reaches 0, an ICMP TTL Exceeded message is returned to the source of the packet. This message can be used to discover the hops between a given source and destination.

With that out of the way, we can move on to the different methods of generating this response from every hop along a given route. Linux comes with a traceroute utility built-in, and this is the tool that I'm going to be investigating. If you're on Windows, you can use tracert, but it doesn't have as many options as the Linux version.

Linux's traceroute utility defaults to using UDP packets on an uncommon port. It defaults to this because it's the best method that unprivileged users can use if they have a kernel older than 3.0 (check your kernel version with uname -r). It isn't ideal though, because many hosts don't expect incoming UDP packets and silently drop them.

Adding the -I flag causes traceroute to use ICMP ping requests instead. Thankfully most hosts will respond to ICMP pings, making it a much better probing tool. Some networks, however, don't allow ping requests to pass through their gateways (usually large institutions and schools), rendering this method useless in certain situations.

To combat the above, a new method was developed that uses TCP SYN packets instead of UDP or ICMP ping. If you send a TCP SYN packet (manipulating the TTL as above), practically all hosts will return some kind of message. This is commonly referred to as the TCP half-open technique, and defaults to port 80 - this allows the traceroute to bypass nearly all firewalls. If you're behind a proxy though I suspect it'll snag on it - theoretically speaking using port 443 instead should rectify this problem in most cases (i.e. traceroute -T -p 443 hostname.tld).

Traceroute has a bunch of other less reliable methods, which I'll explain quickly below.

• -U causes traceroute to use UDP on port 53. This method usually only elicits responses from DNS servers along the route.
• -UL makes traceroute use udplite in a similar fashion to UDP in the bullet point above. This is only available to administrators.
• DCCP can also be used with the -D. It works similar to the TCP method described earlier.
• A raw IP packet can also be used, but I can't think of any reasons you'd use this.

## Demystifying UDP

Yesterday I was taking a look at [UDP Multicast], and attempting to try it out in C#. Unfortunately, I got a little bit confused as to how it worked, and ended up sending a couple of hours wondering what I did wrong. I'm writing this post to hopefully save you the trouble of fiddling around trying to get it to work yourself.

UDP stands for User Datagram Protocol (or Unreliable Datagram Protocol). It offers no guarantee that message sent will be received at the other end, but is usually faster than its counterpart, TCP. Each UDP message has a source and a destination address, a source port, and a destination port.

When you send a message to a multicast address (like the 239.0.0.0/8 range or the FF00::/8 range for ipv6, but that's a little bit more complicated), your router will send a copy of the message to all the other interested hosts on your network, leaving out hosts that have not registered their interest. Note here that an exact copy of the original message is sent to all interested parties. The original source and destination addresses are NOT changed by your router.

With that in mind, we can start to write some code.

IPAddress multicastGroup = IPAddress.Parse("239.1.2.3");
int port = 43;
IPEndPoint channel = new IPEndPoint(multicastGroup, port);
UdpClient client = new UdpClient(43);
client.JoinMulticastGroup(multicastGroup);

In the above, I set up a few variables or things like the multicast address that we are going to join, the port number, and so on. I pass the port number to the new UdpClient I create, letting it know that we are interested in messages sent to that port. I also create a variable called channel, which we will be using later.

Next up, we need to figure out a way to send a message. Unfortunately, the UdpClient class only supports sends arrays of bytes, so we will be have to convert anything we want to send to and from a byte array. Thankfully though this isn't too tough:

string data = "1 2, 1 2, Testing!";
string message = Encoding.UTF8.GetString(payload);

The above converts a simple string to and from a byte[] array. If you're interested, you can also serialise and deserialise C♯ objects to and from a byte[] array by using Binary Serialisation. Anyway, we can now write a method to send a message across the network. Here's what I came up with:

private static void Send(string data)
{
Console.WriteLine("Sending '{0}' to {1}.", data, destination);
}
{
}

Here I've defined a method to send stuff across the network for me. I've added an overload, too, which automatically converts string into byte[] arrays for me.

Putting the above together will result in a multicast message being sent across the network. This won't do us much good though unless we can also receive messages from the network too. Let's fix that:

public static async Task Listen()
{
while(true)
{
string message = Encoding.UTF8.GetString(result.Buffer);
Console.WriteLine("{0}: {1}", result.RemoteEndPoint, message);
}
}

You might not have seen (or heard of) asynchronous C# before, but basically it's a ways of doing another thing whilst you are waiting for one thing to complete. Dot net perls have a good tutorial on the subject if you want to read up on it.

For now though, here's how you call an asynchronous method from a synchronous one (like the Main() method since that once can't be async apparently):

Task.Run(() => Listen).Wait();

If you run the above in one program while sending a message in another, you should see something appear in the console of the listener. If not, your computer may not be configured to receive multicast messages that were sent from itself. In this case try running the listener on a different machine to the sender. In theory you should be able to run the listener on as many hosts on your local network as you want and they should all receive the same message.

## Reading HTTP 1.1 requests from a real web server in C#

I've received rather a lot of questions recently asking the same question, so I thought that I 'd write a blog post on it. Here's the question:

Why does my network client fail to connect when it is using HTTP/1.1?

I encountered this same problem, and after half an hour of debugging I found the problem: It wasn't failing to connect at all, rather it was failing to read the response from the server. Consider the following program:

using System;
using System.IO;
using System.Net.Sockets;

class Program
{
static void Main(string[] args)
{
TcpClient client = new TcpClient("host.name", 80);
client.SendTimeout = 3000;
StreamWriter writer = new StreamWriter(client.GetStream());
writer.WriteLine("GET /path HTTP/1.1");
writer.WriteLine("Host: server.name");
writer.WriteLine();
writer.Flush();

Console.WriteLine("Got Response: '{0}'", response);
}
}

If you change the hostname and request path, and then compile and run it, you (might) get the following error:

An unhandled exception of type 'System.IO.IOException' occurred in System.dll

Additional information: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time or established connection failed because connected host has failed to respond.

Strange. I'm sure that we sent the request. Let's try reading the response line by line:

string response = string.Empty;
do
{
response += nextLine;
Console.WriteLine("> {0}", nextLine);
} while (reader.Peek() != -1);

Here's some example output from my server:

> HTTP/1.1 200 OK
> Server: nginx/1.9.10
> Date: Tue, 09 Feb 2016 15:48:31 GMT
> Content-Type: text/html
> Transfer-Encoding: chunked
> Connection: keep-alive
> Vary: Accept-Encoding
> strict-transport-security: max-age=31536000;
>
> 2ef
> <html>
> <body bgcolor="white">
> <h1>Index of /libraries/</h1><hr><pre><a href="../">../</a>
> <a href="prism-grammars/">prism-grammars/</a>
09-Feb-2016 13:56                   -
> <a href="blazy.js">blazy.js</a>                                           09-F
eb-2016 13:38                9750
> <a href="prism.css">prism.css</a>                                          09-
Feb-2016 13:58               11937
> <a href="prism.js">prism.js</a>                                           09-F
eb-2016 13:58               35218
> <a href="smoothscroll.js">smoothscroll.js</a>
20-Apr-2015 17:01                3240
> </pre><hr></body>
> </html>
>
> 0
>

...but we still get the same error. Why? The reason is that the web server is keeping the connection open, just in case we want to send another request. While this would usually be helpful (say in the case of a web browser - it will probably want to download some images or something after receiving the initial response), it's rather a nuisance for us, since we don't want to send another request and it's rather awkward to detect the end of the response without detecting the end of the stream (that's what the while (reader.Peek() != -1); is for in the example above).

Thankfully, there are a few solutions to this. Firstly, the web server will sometimes (but not always - take the example response above for starters) send a content-length header. This header will tell you how many bytes follow after the double newline (\r\n\r\n) that separate the response headers from the response body. We could use this to detect the end of the message. This is the reccommended way , according to RFC2616.

Another way to cheat here is to send the connection: close header. This instructs the web server to close the connection after sending the message (Note that this will break some of the tests in the ACW, so don't use this method!). Then we can use reader.ReadToEnd() as normal.

A further cheat would be to detect the expected end of the message that we are looking for. For HTML this will practically always be </html>. We can close the connection after we receive this line (although this doesn't work when you're not receiving HTML). This is seriously not a good idea. The HTML could be malformed, and not contain </html>.

Art by Mythdael