Mailing List Articles Atom Feed Comments Atom Feed Twitter Reddit Facebook

Tag Cloud

3d account algorithms announcement architecture archives arduino artificial intelligence artix assembly async audio bash batch blog bookmarklet booting c sharp c++ challenge chrome os code codepen coding conundrums coding conundrums evolved command line compilers compiling compression css dailyprogrammer debugging demystification distributed computing documentation downtime electronics email embedded systems encryption es6 features event experiment external first impressions future game github github gist graphics hardware hardware meetup holiday holidays html html5 html5 canvas infrastructure interfaces internet io.js jabber javascript js bin labs learning library linux low level lua maintenance manjaro network networking node.js operating systems performance photos php pixelbot portable privacy programming problems project projects prolog protocol protocols pseudo 3d python reddit reference release releases resource review rust secrets security series list server software sorting source code control statistics storage svg technical terminal textures three thing game three.js tool tutorial tutorials twitter ubuntu university update updates upgrade version control virtual reality virtualisation visual web website windows windows 10 xmpp xslt

GlidingSquirrel is now on NuGet with automatic API documentation!

(This post is a bit late as I'm rather under the weather.)

A while ago, I posted about a HTTP/1.0 server library I'd written in C♯ for a challenge. One thing led to another, and I've now ended up adding support for HTTP/1.1, WebSockets, and more....

While these enhancements have been driven mainly by a certain project of mine that keeps stealing all of my attention, they've been fun to add - and now that I've (finally!) worked out how to package it as a NuGet package, it's available on NuGet for everyone to use, under the Mozilla Public License 2.0!

Caution should be advised though, as I've not thoroughly tested it yet to weed out the slew of bugs and vulnerabilities I'm sure are hiding in there somewhere - it's designed mainly to sit behind a reverse-proxy such as Nginx (not that's it's any excuse, I know!) - and also I don't have a good set of test cases I can check it against (I thought for sure there would be at least one test suite out there for HTTP servers, considering the age of the protocol, but apparently not!).

With that out of the way, specifying dependencies didn't turn out to be that hard, actually. Here's the extra bit I added to the .nuspec file to specify the GlidingSquirrel's dependencies:

    <dependency id="MimeSharp" version="1.0.0" />
    <dependency id="System.ValueTuple" version="4.4.0" />

The above goes inside the <metadata> element. If you're curious about what this strange .nuspec file is and it's format, I've blogged about it a while back when I figured out how to package my TeleConsole Client - in which I explain my findings and how you can do it yourself too!

I've still not figured out the strange $version$ and $description$ things that are supposed to reference the appropriate values in the .csproj file (if you know how those work, please leave a comment below, as I'd love to know!) - as it keeps telling me that <description> cannot be empty.....

In addition to packaging it and putting it up on NuGet, I've also taken a look at automatic documentation generation. For a while I've been painstakingly documenting the entire project with intellisense comments for especially this purpose - which I'm glad I started early, as it's been a real pain and taken a rather decent chunk of time to do.

Anyway, with the last of the intellisense comments finished today, I set about generating automatic documentation - which turned out to be a surprisingly difficult thing to achieve.

  • Sandcastle, Microsoft's offering has been helpfully discontinued.
  • Sandcastle Help File Builder, the community open-source fork of Sandcastle, appears to be Windows-only (I use Linux myself).
  • DocFX, Microsoft's new offering, appears to be more of a static site generator that uses your code and a bunch of other things as input, rather than the simple HTML API documentation generator I was after.
  • I found Docxygen to produce an acceptable result out-of-the-box, I discovered that configuring it was not a trivial task - the initial configuration file the init action created was over 100 KiB!

I found a few other options too, but they were either Windows-only, or commercial offerings that I can't justify using for an open-source project.

When I was about to give up the search for the day, I stumbled across this page on generating documentation by the mono project, who just happen to be the ones behind mono, the cross-platform .NET runtime that runs on Linux, macOS, and, of course, Windows. They're also the ones who have built mcs, the C♯ compiler that compliments the mono runtime.

Apparently, they've also built a documentation generation that has the ability to export to HTML. While it's certainly nothing fancy, it does look like you've all the power you need at your fingertips to customise the look and feel by tweaking the XSLT stylesheet it renders with, should you need it.

After a bit of research, I found this pair of commands to build the documentation for me quite nicely:

mdoc update -o docs_xml/ -i GlidingSquirrel.xml GlidingSquirrel.dll
mdoc export-html --out docs/ docs_xml

The first command builds the multi-page XML tree from the XML documentation generated during the build process (make sure the "Generate xml documentation" option is ticked in the project build options!), and the second one transforms that into HTML that you can view in your browser.

With the commands figured out, I set about including it in my build process. Here's what I came up with:

<Target Name="AfterBuild">
    <Message Importance="high" Text="----------[ Building documentation ]----------" />

    <Exec Command="mdoc update -o docs_xml/ -i GlidingSquirrel.xml GlidingSquirrel.dll" WorkingDirectory="$(TargetDir)" IgnoreExitCode="true" />
    <Exec Command="mdoc export-html --out $(SolutionDir)/docs docs_xml" WorkingDirectory="$(TargetDir)" IgnoreExitCode="true" />

This outputs the documentation to the docs/ folder in the root of the solution. If you're a bit confused as to how to utilise this in your own project, then perhaps the Untangling MSBuild post I wrote to figure out how MSBuild works can help! You can view the GlidingSquirrel's automatically-generated documentation here. It's nothing fancy (yet), but it certainly does the job. I'll have to look into prettifying it later!

PixelBot Part 2: Devices need protocols, apparently

A selection of the technologies I'm using to put together my PixelHub server.

So there I was. I'd just got home, turned on my laptop, opened the Arduino IDE and Monodevelop, and then.... nothing. I knew I wanted my PixelBot to talk to the PixelHub I'd started writing, but I was confused as to how I could make it happen.

In this kind of situation, I realised that although I knew what I wanted them to do, I hadn't figured out the how. As it happens, when you're trying to get one (or more, in this case) different devices to talk to each other, there's something rather useful that helps them all to speak the same language: a protocol.

A protocol is a specification that defines the language that different devices use to talk to each other and exchange messages. Defining one before you start writing a networked program is probably a good idea - I find particularly helpful to write a specification for the protocol that the program(s) I'm writing, especially if their function(s) is/are complicated.

To this end, I've ended up spending a considerable amount of time drawing up the PixelHub Protocol - a specification document that defines how my PixelHub server is going to talk to a swarm of PixelBots. It might seem strange at first, but I decided on a (mostly) binary protocol.

Upon closer inspection though, (I hope) it makes a lot of sense. Since the Arduino is programmed using C++ as it has a limited amount of memory, it doesn't have any of the standard string manipulation function that you're used to in C♯. Since C++ is undoubtedly the harder of the 2 to write, I decided to make it easier to write the C++ rather than the C&sharp. Messages on the Arduino side are come in as a byte[] array, so (in theory) it should be easy to pick out certain known parts of the array and cast them into various different fundamental types.

With the specification written, the next step in my PixelBot journey is to actually implement it, which I'll be posting about in the next entry in this series!

My PixelBot is Connected! (Part 1)

After many attempts and solving many problems, I've finally gotten my Wemos-powered PixelBot to connect via TCP to a C# server component I've written. Since I experienced so many problems with it, I decided to post about it here to help others who want to do the same as I.

The first (and arguably most difficult) hurdle I came across was the lack of correct information. For one, the TCP client class is actually called WifiClient (which is confusing in and of itself). For another, there aren't any really good tutorials out there that show you what to do.

In addition, I wanted to build a (rather complicated as it turns out!) auto discovery system to allow my PixelBot to find the PixelHub (the server) automatically. As usual, there was even less information about this task! All I found was an outdated guide and rather simplistic example. I ended up inspecting the header file of the WiFiUDP class in order to figure it out.

In this post, I'm going to explain how I got the autodiscovery mechanism working. Before we begin, you need to understand what multicast UDP is and how it works. For those of you who don't, I've posted all about it.

There are several ways that one can go about writing an autodiscovery mechanism. While Rob Miles chose mDNS, I ended up approaching the problem from a different angle and writing my own protocol. It's best explained with a diagram:

A diagram explaining the PixelBot's autodiscovery mechanism.

  1. The PixelHub server has a beacon that sends out pings to tell the PixelBots where it is
  2. The PixelBots subscribe to the beacon ping channel when they want to find the server
  3. The PixelBots decode the incoming ping packets to find the discover of the server
  4. The PixelBots connect to the PixelHub server using the credentials that it found in the beacon pings it decoded!

In order to receive these beacon pings, we need to set up a UDP listener and wire it up to receive multicast packets. In the following example I'm multicasting on on port 5050.

IPAddress beaconAddress = IPAddress(239, 62, 148, 30);
unsigned int beaconPort = 5050;

WiFiUDP UdpClient;
UdpClient.beginMulticast(WiFi.localIP(), beaconAddress, beaconPort);

After connecting to the multicast address, we then need to receive the pings:

// Create a buffer to hold the message 
byte datagramBuffer[datagramBufferSize];
// Prefill the datagram buffer with zeros for protection later
memset(datagramBuffer, '\0', datagramBufferSize);
while(true) {
    int datagramSize = UdpClient.parsePacket();
    Serial.print("Received datagram #");
    Serial.print(" bytes in size from ");

    // Don't overflow the message buffer!
    if(datagramSize > datagramBufferSize) {
        Serial.println(", but the message is larger than the datagram buffer size.");
    // Read the message in now that we've verified that it won't blow our buffer up, datagramSize);

    // Cheat and cast the datagram to a character array
    char* datagramStr = (char*)datagramBuffer;


That's rather complicated for such a simple task! Anyway, now we've got our beacon ping into a character array, we can start to pick it apart. Before we do, here's what my beacon pings look like:

  ^        ^        ^
 Role IP Address   Port

Although it looks simple, in C++ (the language of the arduino) it's rather a pain. Here's what I came up with:

// Define the role of the remote server that we're looking for
char desiredRemoteRole[7] = { 's', 'e', 'r', 'v', 'e', 'r', '\0' };

// Find the positions of the key characters
int atPos = findChar(datagramStr, '@');
int colonPos = findChar(datagramStr, ':');

// Create some variables to store things in
char role[7];
char serverIp[16];
char serverPortText[7];
int serverPort = -1;
// Fill everything with zeroes to protect ourselves
memset(role, '\0', 7);
memset(serverIp, '\0', 16);
memset(serverPortText, '\0', 7);
// Extract the parts of the 
strncpy(role, datagramStr, atPos);
strncpy(serverIp, datagramStr + atPos + 1, colonPos - atPos - 1);
strncpy(serverPortText, datagramStr + colonPos + 1, datagramSize - colonPos - 1);


// Print everything out to the serial console
Serial.print("atPos: "); Serial.println(atPos);
Serial.print("colonPos: "); Serial.println(colonPos);

Serial.print("Role: "); Serial.print(role); Serial.print(" ");
Serial.print("Remote IP: "); Serial.print(serverIp); Serial.print(" ");
Serial.print("Port number: "); Serial.print(serverPortText);

// If the advertiser isn't playing the role of a server, then we're not interested
if(strcmp(role, desiredRemoteRole) != 0)
    Serial.print("Incompatible role "); Serial.print(role); Serial.println(".");
Serial.println("Role ok!");
serverPort = atoi(serverPortText);

Phew! That's a lot of code! I've tried to annotate it the best I can. The important variables in the are serverIp the serverPort - they hold the IP address and the port number of the remote machine that we want to connect to.

I hope that's somewhat helpful. If there's anything you don't understand or need help with, please post a comment down below :-)

Next time, I'm going to show off the TCP connection bit of the system. Stay tuned :D

Pepperminty Wiki Turns 2!

Pepperminty Wiki turns 2(!) :D

2 years ago today, I decided that I'd build a wiki. At the time, I was unsatisfied with all the currently solutions out there - they were either too bulky, old and unmaintained, or just otherwise not quite right. I decided to build something different: An entire wiki in a single file. One single file that you can drop onto your web server and have it just work.

Although I've had to rework and rewrite a few things along the way, development has generally gone ok. I've found that as it's grown, I've needed to change the way I design it slightly - it's been a great learning experience!

In May this year, I managed to get Pepperminty Wiki into Awesome Self Hosted, a list of cool pieces of software that you can host on your own server - a list that has ~12,000 stars on GitHub.

Fast forward to the present, and I find myself with an awesome wiki - that's still all contained in a single file. It's powered by Parsedown. It supports multiple users, page protection, sub pages, full text search (!), customisable themes, tags, redirects, and more! It's also got a whole host of configurable settings system - allowing you to customise how your wiki works to your liking. Best of all, it's got a flexible module based system - so anyone can come along and write a new module to extend it's functionality.

If this sounds like the kind of thing you'd like to use yourself, just head over to the getting your own copy section on it's page (for the lazy, the online downloader is here).

Going forwards, I'd like to a commenting system to let people comment on pages on a wiki. I'd like to add edit previews. I'd like to add a GUI for the settings file. I've got so many ideas that it's difficult to choose which to do next :D

Thank you, everyone for the last 2 years. Here's to another 2 amazing fun filled years! I don't intend to stop development any time soon :D

Pepperminty Wiki is now on WikiMatrix!

The WikiMatrix Logo

After a year of waiting (and hard work improving it), Pepperminty Wiki is now on!

WikiMatrix is a brilliant site that lets you compare many different pieces of wiki software with each other in order to figure out which one best meets your needs. Unfortunately, the admins are rather difficult to get hold of, and so new wikis get added rarely.

If you're looking to set up a a small wiki, I recommend checking out my project Pepperminty Wiki. It's inspired by am2064's logo am2064's Minty Wiki that he posted on /r/tinycode, and is designed to be uploaded to a web server and just work (after configuring it, of course!). It's packed into a single file and currently weighs in at ~160kb, so you won't have to wait around for ages for it to download or unpack.

Notable features include multiple users, subpages, templates, file uploads, a full text search engine, page protection, and more! New features are being added all the time - an up to date list can be found in the Github README (link below). Features can be added or removed at any time, too - Pepperminty Wiki sports a module based approach, letting you decide on what features your wiki has enabled.

Link: Pepperminty Wiki

Pepperminty Wiki

3D Worley Noise with noisebox

Worley Noise Recently, I've been writing a command line noise generation tool in C♯, and I've called it noisebox. Initially I found a few noise generation algorithms that other people had already implemented, so all I had to do was write an extensible interfacde on top of the code I'd found. When I came to Worley noise, I couldn't find an implementation that I could understand (I haven't taken a look at delegates properly yet), so I decided to write my own. It's still got some bugs in it, but I've decided to relase the Worley noise implementation on it's own first, and then I will tidy up a few loose ends and release the full code on GitHub (loko out for a blog post soon!).

Here's a link to a gist of the Worley noise generator: Worley.cs

The imbuilt documentation comments (what do you call them?) should give you enough information to use it, but if you get stuck post a comment below and I will try and help you out.

The code will be released under the Mozilla Public License 2.0.

This post's title include the word "3D" - but the image at the top is very much in 2D. To demonstrate the 3D-ness of the algorithm, I added the --frames and --offset options to noisebox and rendered 1000 frames of noise, and then stitched the together with ffmpeg. I've uploaded the result to youtube.

The Big Warehouse: A collection of Computer Science Related Links


Today I was going to have the second tutorial in the XMPP: A Lost Protocol series, but asciinema, the terminal recording system I use, crashed when I absent mindedly resized my PuTTY window. I will try recording again soon.

Instead, I have a shorter post about a github project that I started a while ago called The Big Warehouse. Sadly it isn't a game but rather a collection of links organised into a number of Big Boxes. Each big box contains a collection of links about a particular programming language or system (e.g. Javascript, CSS, Version Control Systems, etc.).

Every time I find a good tutorial or library, I will add it to the appropriate big box. If you find a cool thing you think would fit in the warehouse, simply open an issue or submit a pull request. If you don't have a github account, comment on this post and I will add it that way (I get notified about all comments on this blog).

You can find it here: The Big Warehouse

Art by Mythdael