Archive

## Tag Cloud

3d account algorithms announcement archives arduino artificial intelligence assembly async audio bash batch blog bookmarklet booting c sharp c++ challenge chrome os code codepen coding conundrums coding conundrums evolved command line compiling css dailyprogrammer debugging demystification distributed computing downtime embedded systems encryption es6 features event experiment external first impressions future game github github gist graphics hardware hardware meetup holiday html html5 html5 canvas interfaces internet io.js jabber javascript js bin labs learning library linux low level lua maintenance network networking node.js operating systems performance photos php pixelbot portable privacy programming problems project projects prolog protocol protocols pseudo 3d python reddit reference release releases resource review rust secrets security series list server servers software sorting source code control statistics svg technical terminal textures three thing game three.js tool tutorial tutorials twitter ubuntu university update updates upgrade version control visual web website windows windows 10 xmpp

## PixelBot Part 2: Devices need protocols, apparently

So there I was. I'd just got home, turned on my laptop, opened the Arduino IDE and Monodevelop, and then.... nothing. I knew I wanted my PixelBot to talk to the PixelHub I'd started writing, but I was confused as to how I could make it happen.

In this kind of situation, I realised that although I knew what I wanted them to do, I hadn't figured out the how. As it happens, when you're trying to get one (or more, in this case) different devices to talk to each other, there's something rather useful that helps them all to speak the same language: a protocol.

A protocol is a specification that defines the language that different devices use to talk to each other and exchange messages. Defining one before you start writing a networked program is probably a good idea - I find particularly helpful to write a specification for the protocol that the program(s) I'm writing, especially if their function(s) is/are complicated.

To this end, I've ended up spending a considerable amount of time drawing up the PixelHub Protocol - a specification document that defines how my PixelHub server is going to talk to a swarm of PixelBots. It might seem strange at first, but I decided on a (mostly) binary protocol.

Upon closer inspection though, (I hope) it makes a lot of sense. Since the Arduino is programmed using C++ as it has a limited amount of memory, it doesn't have any of the standard string manipulation function that you're used to in C♯. Since C++ is undoubtedly the harder of the 2 to write, I decided to make it easier to write the C++ rather than the C&sharp. Messages on the Arduino side are come in as a byte[] array, so (in theory) it should be easy to pick out certain known parts of the array and cast them into various different fundamental types.

With the specification written, the next step in my PixelBot journey is to actually implement it, which I'll be posting about in the next entry in this series!

## Easier TCP Networking in C♯

I see all sorts of C♯ networking tutorials out there telling you that you have to use byte arrays and buffers and all sorts of other complicated things if you ever want to talk to another machine over the network. Frankly, it's all rather confusing.

Thankfully though, it doesn't have to stay this way. I've learnt a different way of doing TCP networking in C♯ at University (thanks Brian!), and I realised the other day I've never actually written a blog post about it (that I can remember, anyway!). If you know how to read and write files and understand some basic networking concepts (IP addresses, ports, what TCP and UDP are, etc.), you'll have no problems understanding this.

### Server

The easiest way to explain it is to demonstrate. Let's build a quick server / client program where the server says hello to the client. Here's the server code:


// Server.cs
using System;
using System.Net;
using System.Net.Sockets;
using System.IO;

public class Server
{
public Server(int inPort)
{
Port = inPort; string s;
}

{
TcpListener server = new TcpListener(IPAddress.Any, Port);
server.Start();
while (true) {
TcpClient nextClient = await server.AcceptTcpClientAsync();
StreamWriter outgoing = new StreamWriter(nextClient.GetStream()) { AutoFlush = true };

await outgoing.WriteLineAsync($"Hello, {name}!"); Console.WriteLine("Said hello to {0}", name); nextClient.Close(); } } } // Use it like this in your Main() method: Server server = new Server(6666); server.Start().Wait();  Technically speaking, that asynchronous code ought to be running in a separate thread - I've omitted it to make it slightly simpler :-) Let's break this down. The important bit is in the Start() method - the rest is just sugar around it to make it run if you want to copy and paste it. First, we create & start a TcpListener: TcpListener server = new TcpListener(IPAddress.Any, Port); server.Start(); Once done, we enter a loop, and wait for the next client: TcpClient nextClient = await server.AcceptTcpClientAsync(); Now that we have a client to talk to, we attach a StreamReader and a StreamWriter with a special option set on it to allow us to talk to the remote client with ease. The option set on the StreamWriter is AutoFlush, and it basically tells it to flush it's internal buffer every time we write to it - that way things we write to it always hit the TcpClient underneath. Depending on your setup the TcpClient does some internal buffering & optimisations anyway, so we don't need the second layer of buffering here: StreamReader incoming = new StreamReader(nextClient.GetStream()); StreamWriter outgoing = new StreamWriter(nextClient.GetStream()) { AutoFlush = true }; With that, the rest should be fairly simple to understand: string name = (await incoming.ReadLineAsync()).Trim(); await outgoing.WriteLineAsync($"Hello, {name}!");

Console.WriteLine("Said hello to {0}", name);

nextClient.Close();

First, we grab the first line that the client sends us, and trim of any whitespace that's lurking around. Then, we send back a friendly hello message to client, before logging what we've done to the console and closing the connection.

### Client

Now that you've seen the server code, the client code should be fairly self explanatory. The important lines are highlighted:


using System;
using System.Net;
using System.Net.Sockets;
using System.IO;

public class Client
{

public Client(string inHostname, int inPort)
{
Hostname = inHostname; uint a;
Port = inPort;
}

public async Task GetHello(string name) {
TcpClient client = new TcpClient();
client.Connect(Hostname, Port);

StreamWriter outgoing = new StreamWriter(client.GetStream()) { AutoFlush = true };

await outgoing.WriteLineAsync(name);

}
}

// Use it like this in your Main() method:
Client client = new Client("localhost", 6666);


First, we create a new client and connect it to the server. Next, we connect the StreamReader and StreamWriter instances to the TcpClient, and then we send the name to the server. Finally, we read the response the server sent us and return it. Easy!

Here's some example outputs:

Client:


./NetworkingDemo-Server.exe
Said hello to Bill


Server:


./NetworkingDemo-Client.exe
The server said: Hello, Bill!


The above code should work on Mac, Windows, and Linux. Granted, it's not the most efficient way of doing things, but it should be fine for most general purposes. Personally, I think the trade-off between performance and readability/ease of understanding of code is totally worth it.

If you prefer, I've got an archive of the above code I wrote for this blog post - complete with binaries. You can find it here: NetworkingDemo.7z.

## Picking the right interface for multicast communications

At the recent hardware meetup, I was faced with an interesting problem: I was trying to communicate with my Wemos over multicast UDP in order to get it to automatically discover my PixelHub Server, but the multicast pings were going out over the wrong interface - I had both an ethernet cable plugged in and a WiFi hotspot running on my integrated wireless card.

The solution to this is not as simple you might think - you have to not only pick the right interface, but also the right version of the IP protocol. You also have to have some way to picking the correct interface in the first place.

Let's say you have a big beefy PC with a wireless card and 2 ethernet ports that are (for some magical reason) all in use at the same time, and you want to communicate with another device over your wireless card and not either of your ethernet ports.

I developed this on my linux laptop, but it should work just fine on other OSes.

To start, it's probably a good idea to list all of our network interfaces:

using System.Net.NetworkInformation;

// ...

NetworkInterface[] nics = NetworkInterface.GetAllNetworkInterfaces();
foreach (NetworkInterface nic in nics)
{
Console.WriteLine("Id: {0} - Description: {1}", nic.Id, nic.Description);
}

This (on my machine at least!) outputs something like this:

eth0 - eth0
lo - lo
wlan0 - wlan0

Your machine will probably output something different. Next, since you can't normally address this list of network interfaces directly by name, we need to write a method to do it for us:

public static NetworkInterface GetNetworkIndexByName4(string targetInterfaceName)
{
NetworkInterface[] nics = NetworkInterface.GetAllNetworkInterfaces();
foreach (NetworkInterface nic in nics)
{
if (nic.Id == targetInterfaceName)
return nic;
}
throw new Exception(\$"Error: Can't find network interface with the name {targetInterfaceName}.");
}

Pretty simple, right? We're not out the woods yet though - next we need to tell our UdpClient to talk on a specific network interface. Speaking of which, let's set up that UdpClient so that we can use it to do stuff with multicast:

using System.Net;

// ...

UdpClient client = new UdpClient(5050);
client.JoinMulticastGroup(IPAddress.Parse("239.62.148.30"));

With that out of the way, we can now deal with telling the UcpClient which network interface it should be talking on. This is actually quite tricky, since the UdpClient doesn't take a NetworkInterface directly. Let's define another helper method:

public static int GetIPv4Index(this NetworkInterface nic)
{
IPInterfaceProperties ipProps = nic.GetIPProperties();
IPv4InterfaceProperties ip4Props = ipProps.GetIPv4Properties();
return ip4Props.Index;
}

The above extension method gets the index of the IPv4 interface of the network interface. Since at the moment we are in the middle of a (frustratingly slow) transition from IPv4 and IPv6, each network interface must have both an IPv4 interface, for talking to other IPv4 hosts, and an IPv6 interface for talking to IPv6 hosts. In this example I'm using IPv4, since the Wemos I want to talk to doesn't support IPv6 :-(

Now that we have a way to get the index of a network interface, we need to translate it into something that the UdpClient understands:

int interfaceIndex = (int)IPAddress.HostToNetworkOrder(NetTools.GetNetworkIndexByName4(NetworkInterfaceName));

That complicated! Thankfully, we don't need to pick it apart completely - it just works :-)

Now that we have the interface index in the right format, all we have to do is tell the UdpClient about it. Again, this is also slightly overcomplicated:

beacon.Client.SetSocketOption(
SocketOptionLevel.IP,
SocketOptionName.MulticastInterface,
interfaceIndex
);

Make sure that you put this call before you join the multicast group. With that done, your UdpClient should finally be talking on the right interface!

Whew! That was quite the rabbit hole (I sent my regards to the rabbit :P). If you have any issues with getting it to work, I'm happy to help - just post a comment down below.

### Sources

• Openclipart images: 1, 2

## Creating a HTTP Server in C♯

(Image from here.)

I discovered the HttpListener class in .NET the other day, and decided to take a look. It turns out that it's way easier to use than doing it all by hand as I had to do in my 08241 Networking module a few months ago. I thought I'd write a post about it so that you can see how easy it is and use the same technique yourself. You might want to add a HTTP status reporting system to your program, or use it to serve a front-end GUI. You could even build a real-time app with WebSockets.

As with all my tutorials - this post is a starting point, not an ending point. Refactor it to your heart's content!

To start off, let's create a new C♯ console project and create an instance of it:

using System;
using System.Net;

class MainClass
{
public static void Main(string[] args)
{
HttpListener listener = new HttpListener();
}
}

This should look pretty familiar to you. If not, then I can recommend the C♯ yellow book by Rob Miles. This is all rather boring so far, so let's move on quickly.

listener.Prefixes.Add("http://*:3333/");
listener.Start();

C♯'s HttpListener works on prefixes. It parses the prefix to work out what it should listen on and for what. The above listens for HTTP requests from anyone on port 3333. Apparently you can listen for HTTPS requests using this class too, but I'd recommend putting it behind some kind of proxy like NginX instead and let that handle the TLS certificates instead.

We also start the listener listening for requests too, because it doesn't listen for requests by default.


while(true)
{
HttpListenerContext cycle = listener.GetContext();
Console.WriteLine("Got request for {0} from {1}.", cycle.Request.RawUrl, cycle.Request.RemoteEndPoint);
StreamWriter outgoing = new StreamWriter(cycle.Response.OutputStream);
}


This next part is the beginning of the main request / response loop. The highlighted line is the important one - it retrieves the next pending request, waiting for it if necessary. The following lines log to the console about the latest request, and set up a StreamWriter to make it easier to send the response to the requester.

The final piece of the puzzle actually sending the request and moving onto the next one.

outgoing.WriteLine("It works!");
outgoing.Close();
cycle.Response.Close();

Actually writing the response is easy - all you have to do is write it to the StreamWriter. Then all you have to do is call Close() on the stream writer and the Response object, and you're done! Here's the whole thing:

using System;
using System.Net;
using System.IO;

namespace HttpServerTest
{
class MainClass
{
public static void Main(string[] args)
{
HttpListener listener = new HttpListener();
listener.Start();

while(true)
{
HttpListenerContext cycle = listener.GetContext();
Console.WriteLine("Got request for {0} from {1}.", cycle.Request.RawUrl, cycle.Request.RemoteEndPoint);
StreamWriter outgoing = new StreamWriter(cycle.Response.OutputStream);
cycle.Response.ContentType = "text/plain";
outgoing.WriteLine("It works!");
outgoing.Close();
cycle.Response.Close();
}
}
}
}


(Hastebin, Raw)

That's all you really need to create a simple HTTP server in C♯. You should extend this by putting everything that could go wrong in a try..catch statement to prevent the server from crashing. You could also write a simple find / replace templating system to make it easier to serve files from your new HTTP server. Refactoring the above to make it asynchronous wouldn't be a bad idea either.

Have you been building anything interesting recently? Post about in the comments below!

## TeleConsole: A simple remote debugging solution for awkward situations

Several times in the last few months I've found myself in some kind of awkward situation where I need to debug a C♯ program, but the program in question either doesn't have a console, or is on a remote machine. In an ideal world, I'd like to have all my debugging message sent to my development machine for inspection. That way I don't have to check the console of multiple different machines just to get an idea as to what has gone wrong.

C♯ already has System.Diagnostics.Debug, which functions similarly to the Console class, except that it sends data to the Application output window. This is brilliant for things running on your local machine through Visual Studio or MonoDevelop, but not so great when you've got a program that utilises the network and has to run on separate computers. Visual Studio for one starts to freeze up if you open the exact same repository on a network drive that's already open somewhere else.

It is for these reasons that I finally decided to sit down and write TeleConsole. It's a simple remote console that you can use in any project you like. It's split into 2 parts: the client library and the server binary. The server runs on your development machine, and listens for connections from the client library, which you reference in your latest and greatest project and use to send messages to the server.

Take a look here: sbrl/TeleConsole (GitHub) (Direct link to the latest release)

The client API is fully documented with intellisense comments, so I assume it should be very easy to work out how to use it (if there's something I need to do to let you use the intellisense comments when writing your own programs, let me know!). If you need some code to look at, I've created an example whilst testing it.

Although it's certainly not done yet, I'll be definitely be building upon it in the future to improve it and add additional features.

## Demystifying traceroute

(Image from labnol. View the full map at submarinecablemap.com.)

A little while ago someone I know seemed a little bit confused as to how a traceroute interacts with a firewall, so I decided to properly look into and write this post.

Traceroute is the practice of sending particular packets across a network in order to discover all the different hops between the source computer and the destination. For example, here's a traceroute between starbeamrainbowlabs.com and bbc.co.uk:

traceroute to bbc.co.uk (212.58.244.23), 30 hops max, 60 byte packets
1  125.ip-37-187-66.eu (37.187.66.125)  0.100 ms  0.015 ms  0.011 ms
2  be10-147.rbx-g1-a9.fr.eu (37.187.231.169)  0.922 ms  0.912 ms  0.957 ms
3  be100-1187.ldn-1-a9.uk.eu (91.121.128.87)  7.536 ms  7.538 ms  7.535 ms
4  * * *
5  ae-1-3104.ear2.London2.Level3.net (4.69.143.190)  18.481 ms  18.676 ms  18.903 ms
6  unknown.Level3.net (212.187.139.230)  10.725 ms  10.434 ms  10.415 ms
7  * * *
8  ae0.er01.telhc.bbc.co.uk (132.185.254.109)  10.565 ms  10.666 ms  10.603 ms
9  132.185.255.148 (132.185.255.148)  12.123 ms  11.781 ms  11.529 ms
10  212.58.244.23 (212.58.244.23)  10.596 ms  10.587 ms  65.243 ms

As you can see, there are quite a number of hops between us and the BBC, not all of which responded to attempts to probe them. Before we can speculate as to why, it's important to understand how a traceroute is performed.

There are actually a number of different methods to perform a traceroute, but they all have a few things in common. The basic idea exploits something called time to live (TTL). This is a special value that all IP packets have (located 16 bytes into an ipv4 header, and 7 bytes into an ipv6 header for those who are curious) that determines the maximum number of hops that a packet is allowed to go through before it is dropped. Every hop along a packet's route decreases this value by 1. When it reaches 0, an ICMP TTL Exceeded message is returned to the source of the packet. This message can be used to discover the hops between a given source and destination.

With that out of the way, we can move on to the different methods of generating this response from every hop along a given route. Linux comes with a traceroute utility built-in, and this is the tool that I'm going to be investigating. If you're on Windows, you can use tracert, but it doesn't have as many options as the Linux version.

Linux's traceroute utility defaults to using UDP packets on an uncommon port. It defaults to this because it's the best method that unprivileged users can use if they have a kernel older than 3.0 (check your kernel version with uname -r). It isn't ideal though, because many hosts don't expect incoming UDP packets and silently drop them.

Adding the -I flag causes traceroute to use ICMP ping requests instead. Thankfully most hosts will respond to ICMP pings, making it a much better probing tool. Some networks, however, don't allow ping requests to pass through their gateways (usually large institutions and schools), rendering this method useless in certain situations.

To combat the above, a new method was developed that uses TCP SYN packets instead of UDP or ICMP ping. If you send a TCP SYN packet (manipulating the TTL as above), practically all hosts will return some kind of message. This is commonly referred to as the TCP half-open technique, and defaults to port 80 - this allows the traceroute to bypass nearly all firewalls. If you're behind a proxy though I suspect it'll snag on it - theoretically speaking using port 443 instead should rectify this problem in most cases (i.e. traceroute -T -p 443 hostname.tld).

Traceroute has a bunch of other less reliable methods, which I'll explain quickly below.

• -U causes traceroute to use UDP on port 53. This method usually only elicits responses from DNS servers along the route.
• -UL makes traceroute use udplite in a similar fashion to UDP in the bullet point above. This is only available to administrators.
• DCCP can also be used with the -D. It works similar to the TCP method described earlier.
• A raw IP packet can also be used, but I can't think of any reasons you'd use this.

## Demystifying UDP

Yesterday I was taking a look at [UDP Multicast], and attempting to try it out in C#. Unfortunately, I got a little bit confused as to how it worked, and ended up sending a couple of hours wondering what I did wrong. I'm writing this post to hopefully save you the trouble of fiddling around trying to get it to work yourself.

UDP stands for User Datagram Protocol (or Unreliable Datagram Protocol). It offers no guarantee that message sent will be received at the other end, but is usually faster than its counterpart, TCP. Each UDP message has a source and a destination address, a source port, and a destination port.

When you send a message to a multicast address (like the 239.0.0.0/8 range or the FF00::/8 range for ipv6, but that's a little bit more complicated), your router will send a copy of the message to all the other interested hosts on your network, leaving out hosts that have not registered their interest. Note here that an exact copy of the original message is sent to all interested parties. The original source and destination addresses are NOT changed by your router.

With that in mind, we can start to write some code.

IPAddress multicastGroup = IPAddress.Parse("239.1.2.3");
int port = 43;
IPEndPoint channel = new IPEndPoint(multicastGroup, port);
UdpClient client = new UdpClient(43);
client.JoinMulticastGroup(multicastGroup);

In the above, I set up a few variables or things like the multicast address that we are going to join, the port number, and so on. I pass the port number to the new UdpClient I create, letting it know that we are interested in messages sent to that port. I also create a variable called channel, which we will be using later.

Next up, we need to figure out a way to send a message. Unfortunately, the UdpClient class only supports sends arrays of bytes, so we will be have to convert anything we want to send to and from a byte array. Thankfully though this isn't too tough:

string data = "1 2, 1 2, Testing!";
string message = Encoding.UTF8.GetString(payload);

The above converts a simple string to and from a byte[] array. If you're interested, you can also serialise and deserialise C♯ objects to and from a byte[] array by using Binary Serialisation. Anyway, we can now write a method to send a message across the network. Here's what I came up with:

private static void Send(string data)
{
Console.WriteLine("Sending '{0}' to {1}.", data, destination);
}
{
}

Here I've defined a method to send stuff across the network for me. I've added an overload, too, which automatically converts string into byte[] arrays for me.

Putting the above together will result in a multicast message being sent across the network. This won't do us much good though unless we can also receive messages from the network too. Let's fix that:

public static async Task Listen()
{
while(true)
{
string message = Encoding.UTF8.GetString(result.Buffer);
Console.WriteLine("{0}: {1}", result.RemoteEndPoint, message);
}
}

You might not have seen (or heard of) asynchronous C# before, but basically it's a ways of doing another thing whilst you are waiting for one thing to complete. Dot net perls have a good tutorial on the subject if you want to read up on it.

For now though, here's how you call an asynchronous method from a synchronous one (like the Main() method since that once can't be async apparently):

Task.Run(() => Listen).Wait();

If you run the above in one program while sending a message in another, you should see something appear in the console of the listener. If not, your computer may not be configured to receive multicast messages that were sent from itself. In this case try running the listener on a different machine to the sender. In theory you should be able to run the listener on as many hosts on your local network as you want and they should all receive the same message.

## Reading HTTP 1.1 requests from a real web server in C#

I've received rather a lot of questions recently asking the same question, so I thought that I 'd write a blog post on it. Here's the question:

Why does my network client fail to connect when it is using HTTP/1.1?

I encountered this same problem, and after half an hour of debugging I found the problem: It wasn't failing to connect at all, rather it was failing to read the response from the server. Consider the following program:

using System;
using System.IO;
using System.Net.Sockets;

class Program
{
static void Main(string[] args)
{
TcpClient client = new TcpClient("host.name", 80);
client.SendTimeout = 3000;
StreamWriter writer = new StreamWriter(client.GetStream());
writer.WriteLine("GET /path HTTP/1.1");
writer.WriteLine("Host: server.name");
writer.WriteLine();
writer.Flush();

Console.WriteLine("Got Response: '{0}'", response);
}
}

If you change the hostname and request path, and then compile and run it, you (might) get the following error:

An unhandled exception of type 'System.IO.IOException' occurred in System.dll

Additional information: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time or established connection failed because connected host has failed to respond.

Strange. I'm sure that we sent the request. Let's try reading the response line by line:

string response = string.Empty;
do
{
response += nextLine;
Console.WriteLine("> {0}", nextLine);
} while (reader.Peek() != -1);

Here's some example output from my server:

> HTTP/1.1 200 OK
> Server: nginx/1.9.10
> Date: Tue, 09 Feb 2016 15:48:31 GMT
> Content-Type: text/html
> Transfer-Encoding: chunked
> Connection: keep-alive
> Vary: Accept-Encoding
> strict-transport-security: max-age=31536000;
>
> 2ef
> <html>
> <body bgcolor="white">
> <h1>Index of /libraries/</h1><hr><pre><a href="../">../</a>
> <a href="prism-grammars/">prism-grammars/</a>
09-Feb-2016 13:56                   -
> <a href="blazy.js">blazy.js</a>                                           09-F
eb-2016 13:38                9750
> <a href="prism.css">prism.css</a>                                          09-
Feb-2016 13:58               11937
> <a href="prism.js">prism.js</a>                                           09-F
eb-2016 13:58               35218
> <a href="smoothscroll.js">smoothscroll.js</a>
20-Apr-2015 17:01                3240
> </pre><hr></body>
> </html>
>
> 0
>

...but we still get the same error. Why? The reason is that the web server is keeping the connection open, just in case we want to send another request. While this would usually be helpful (say in the case of a web browser - it will probably want to download some images or something after receiving the initial response), it's rather a nuisance for us, since we don't want to send another request and it's rather awkward to detect the end of the response without detecting the end of the stream (that's what the while (reader.Peek() != -1); is for in the example above).

Thankfully, there are a few solutions to this. Firstly, the web server will sometimes (but not always - take the example response above for starters) send a content-length header. This header will tell you how many bytes follow after the double newline (\r\n\r\n) that separate the response headers from the response body. We could use this to detect the end of the message. This is the reccommended way , according to RFC2616.

Another way to cheat here is to send the connection: close header. This instructs the web server to close the connection after sending the message (Note that this will break some of the tests in the ACW, so don't use this method!). Then we can use reader.ReadToEnd() as normal.

A further cheat would be to detect the expected end of the message that we are looking for. For HTML this will practically always be </html>. We can close the connection after we receive this line (although this doesn't work when you're not receiving HTML). This is seriously not a good idea. The HTML could be malformed, and not contain </html>.

Art by Mythdael