Starbeamrainbowlabs

Stardust
Blog


Archive


Mailing List Articles Atom Feed Comments Atom Feed Twitter Reddit Facebook

Tag Cloud

3d 3d printing account algorithms android announcement architecture archives arduino artificial intelligence artix assembly async audio automation backups bash batch blender blog bookmarklet booting bug hunting c sharp c++ challenge chrome os cluster code codepen coding conundrums coding conundrums evolved command line compilers compiling compression conference conferences containerisation css dailyprogrammer data analysis debugging defining ai demystification distributed computing dns docker documentation downtime electronics email embedded systems encryption es6 features ethics event experiment external first impressions freeside future game github github gist gitlab graphics guide hardware hardware meetup holiday holidays html html5 html5 canvas infrastructure interfaces internet interoperability io.js jabber jam javascript js bin labs latex learning library linux lora low level lua maintenance manjaro minetest network networking nibriboard node.js open source operating systems optimisation outreach own your code pepperminty wiki performance phd photos php pixelbot portable privacy problem solving programming problems project projects prolog protocol protocols pseudo 3d python reddit redis reference release releases rendering research resource review rust searching secrets security series list server software sorting source code control statistics storage svg systemquery talks technical terminal textures thoughts three thing game three.js tool tutorial twitter ubuntu university update updates upgrade version control virtual reality virtualisation visual web website windows windows 10 worldeditadditions xmpp xslt

Set up your own Git server with Go Git Service

Go Git Service

Recently I've been finding myself with several private codebases (University ACWs and such) that I've wanted to work on in several places at different times, and that I've also wanted to backup in case of emergency. Git, along with the cloud, were naturally my first choice. At the time, GitHub only offered 5 free private repositories to students, so I started looking around at few different self hosted solutions.

I found software like GitLab and GitList, but the one I found that best suited my needs was Go Git Service. GitLab in particular looked really cool, but it has rather steep minimum requirements that I can't meet.

An example diff view from Go Git Service.

Go Git Service has low minimum requirements, supports multiple users, and allows unlimited private repositories. It even has a forking system that's based on GitHub. If that wasn't enough, the icing on the cake is that it's so ridiculously easy to set up. In fact it's so simple I managed to set a fully working git server up (with all the extras) in just half an hour.

If anyone would like a full tutorial on how to set up Go Git Service, I'll gladly write one up and post it here. Let me know in the comments!

3 Things to do before you upgrade to Ubuntu 16

Since Ubuntu 16.04 long term support has been released, I naturally have been wanting to upgrade to it. However, due to one thing or another, it hasn't been going so well. I decided to put this post together so that you don't make the same mistakes I did. Most of these tips I'm going to share with you in this post can be applied to be major upgrade process, although I found them useful when upgrading my Laptop to Ubuntu 16.04.

Make a backup. Seriously. I can't stress this enough. If the upgrade process gets interrupted or any reason, you'll be left with an unstable system - and a restore will be practically your only option.

Personally I use Timeshift to keep snapshots of my operating system files, and the default Déjà Dup backup program to backup my files. That way I can restore my OS to an earlier state without touching my files at all. Similarly, I can search though the history of my files without the clutter of a bunch of system files getting in the way.

Run the upgrader in either screen or tmux. Screen or Tmux let you run commands in a virtual window, and reattach to that window later if you loose your connection unexpectedly. In my case, I managed to lose access to the upgrader script halfway through the upgrade process, forcing me to kill the process and start over.

Make a checklist of things to do / test after the upgrade. Whilst upgrading, you'll find you start building up a long list of things you want to do and test after the upgrade is finished, such as reactivating extra PPAs, or merging config files. I recommend you write them all down somewhere so that you don't forget something.

That's about all I can think of at the moment. What upgrade are you in the process of doing? Do you have any tips to share? Write a comment below!

Resources

Demystifying UDP

Yesterday I was taking a look at [UDP Multicast], and attempting to try it out in C#. Unfortunately, I got a little bit confused as to how it worked, and ended up sending a couple of hours wondering what I did wrong. I'm writing this post to hopefully save you the trouble of fiddling around trying to get it to work yourself.

UDP stands for User Datagram Protocol (or Unreliable Datagram Protocol). It offers no guarantee that message sent will be received at the other end, but is usually faster than its counterpart, TCP. Each UDP message has a source and a destination address, a source port, and a destination port.

When you send a message to a multicast address (like the 239.0.0.0/8 range or the FF00::/8 range for ipv6, but that's a little bit more complicated), your router will send a copy of the message to all the other interested hosts on your network, leaving out hosts that have not registered their interest. Note here that an exact copy of the original message is sent to all interested parties. The original source and destination addresses are NOT changed by your router.

With that in mind, we can start to write some code.

IPAddress multicastGroup = IPAddress.Parse("239.1.2.3");
int port = 43;
IPEndPoint channel = new IPEndPoint(multicastGroup, port);
UdpClient client = new UdpClient(43);
client.JoinMulticastGroup(multicastGroup);

In the above, I set up a few variables or things like the multicast address that we are going to join, the port number, and so on. I pass the port number to the new UdpClient I create, letting it know that we are interested in messages sent to that port. I also create a variable called channel, which we will be using later.

Next up, we need to figure out a way to send a message. Unfortunately, the UdpClient class only supports sends arrays of bytes, so we will be have to convert anything we want to send to and from a byte array. Thankfully though this isn't too tough:

string data = "1 2, 1 2, Testing!";
byte[] payload = Encoding.UTF8.GetBytes(data);
string message = Encoding.UTF8.GetString(payload);

The above converts a simple string to and from a byte[] array. If you're interested, you can also serialise and deserialise C♯ objects to and from a byte[] array by using Binary Serialisation. Anyway, we can now write a method to send a message across the network. Here's what I came up with:

private static void Send(string data)
{
    Console.WriteLine("Sending '{0}' to {1}.", data, destination);
    byte[] payload = Encoding.UTF8.GetBytes(data);
    Send(payload);
}
private static void Send(byte[] payload)
{
    client.Send(payload, payload.Length, channel);
}

Here I've defined a method to send stuff across the network for me. I've added an overload, too, which automatically converts string into byte[] arrays for me.

Putting the above together will result in a multicast message being sent across the network. This won't do us much good though unless we can also receive messages from the network too. Let's fix that:

public static async Task Listen()
{
    while(true)
    {
        UdpReceiveResult result = await client.ReceiveAsync();
        string message = Encoding.UTF8.GetString(result.Buffer);
        Console.WriteLine("{0}: {1}", result.RemoteEndPoint, message);
    }
}

You might not have seen (or heard of) asynchronous C# before, but basically it's a ways of doing another thing whilst you are waiting for one thing to complete. Dot net perls have a good tutorial on the subject if you want to read up on it.

For now though, here's how you call an asynchronous method from a synchronous one (like the Main() method since that once can't be async apparently):

Task.Run(() => Listen).Wait();

If you run the above in one program while sending a message in another, you should see something appear in the console of the listener. If not, your computer may not be configured to receive multicast messages that were sent from itself. In this case try running the listener on a different machine to the sender. In theory you should be able to run the listener on as many hosts on your local network as you want and they should all receive the same message.

Random Number Generation: The what, why and how

Many computer scientists are absolutely crazy about random numbers. On first thought it sounds a little bit odd, but upon further inspection it's easy to see why. You can use them for generating random loot in a game, or making a monster walk around randomly. Or in cryptography. The possibilities are endless! In addition, sometimes it is desirable to repeat a particular sequence of numbers without storing them all, and sometimes the opposite is true. In this post I intend to explain what a pseudo-random number generator is, why you'd want one, and where you can get your own.

Unfortunately, although computers are really good at complicated calculations, they are totally rubbish at generating true random numbers (that's what random.org is for). All is not lost though - we can still generate long sequences of numbers using a pseudo-random number generator (PRNG).

There are many different algorithms in several different families, but they all rely on a few basic principles. All PRNGs start with a seed, do something to transform the seed, and produce an output. Some algorithms store a few of the previously generated numbers to feed them back into the algorithm too. All PRNGs also have a period, which is the number of random values they can produce before they start to repeat themselves. Usually this value is so high that it doesn't mattter.

The seed, in this case, is a value that is used to initialise the random number generator and give it something to work with. Because PRNGs aren't truly random, any given seed will always produce the same sequence of random numbers. This can be useful if you want to allow players of your game to share cool maps that they've found without having to store details of every single item.

PRNGs can also be measured in terms of the 'quality' of the random numbers they produce. This sounds like a difficult thing to measure, and it is. The easiest (but probably not the best - I don't know, I'm not an expert!) way to test the quality of a random number generator is to generate a whole bunch of random bytes, save them to a file, and try to compress it. A true sequence of random numbers should be uncompressible. Besides, nobody wants a poor-quality random number generator.

After all that, we can finally get around to the algorithms themselves. there are 3 categories that I know of:

Linear Congruential Generators (LCGs)

Your bog standard (poor quality) generator. Usually has relatively short period too. The only good thing about these is their speed.

Mersenne Twister

Slower, but have an insanely large period (2219937 − 1 to be exact). Output is of a high quality.

XOR bitshifters

A family of fast, high-quality generators. Variable period, depending on the algorithm you pick. Also very easy to implement. See below for more information.

There may be others, but these are the 3 that I've seen around (suggestions of algorithm families are welcome in the comments).

Since XOR bit shifters are the new up-and-coming thing, I'll elaborate on them a little bit. There are actually a bunch of different algorithms in this family:

  • xorshift
  • xorshift1024*
  • xorshift128+
  • xoroshiro128+
  • ....

Their history is a bit complicated, so I won't go into any detail, but basically it all started with the xorshift algorithm. The xorshift* family followed as an improvement afterwards, and the xorshift+ family is the result of another (but different) improvement made by tweaking the original algorithm slightly. Finally, the xoroshiro+ set are new and include yet another improvement based on xorshiro+. This article sums them up nicely, along with a suggestion as to when you should use each.

The number in the names of the above refer to the number of bits that each uses to store its state. Apparently each algorithm is available in 64, 128, 256, 1024, and probably even more flavours, but the above are the most popular.

Implementations

To end this post, I'm going to include some links to implementations of the algorithms mentioned in this post in various languages. This (certainly) isn't an exhaustive list, but should serve as a good starting point if you are on the hunt for a random number generator for your next project.

Sources

  1. The multiply-with-carry (aka MWC) algorithm apparently come under xor bitshifters, but I' haven't mentioned it to as to keep things (relatively) simple. More information can be found here.

Update 5th Jan 2017: Fixed a typo.

An introduction to L Systems

Recently I've been taking a look at L Systems. Apparently, they are used for procedural content generation. After playing around with them for a little while, I discovered that you can create some rather cool patterns with them, like this Sierpinski Triangle for instance.

A Sierpinski Triangle

Before we get into how I made the above triangle grow, it's important to understand how an L System works first. The best way to describe an L System is to show you one. Let's start with a single letter:

f

Not very interesting, is it? Let's run a few find and replace rules over it. Here are a few I found lying around:

h=f+h+f f=h-f-h

After running those, here's what we got back:

h-f-h

Hrm. Interesting. What happens if I do it again?

f+h+f-h-f-h-f+h+f

Ah. Now we're getting somewhere. Here are the next 2 runs for reference:

Run 3:

h-f-h+f+h+f+h-f-h-f+h+f-h-f-h-f+h+f-h-f-h+f+h+f+h-f-h

Run 4:

f+h+f-h-f-h-f+h+f+h-f-h+f+h+f+h-f-h+f+h+f-h-f-h-f+h+f-h-f-h+f+h+f+h-f-h-f+h+f-h-f-h-f+h+f-h-f-h+f+h+f+h-f-h-f+h+f-h-f-h-f+h+f+h-f-h+f+h+f+h-f-h+f+h+f-h-f-h-f+h+f

Now that we have run it a few times, it's started to really blow up in size. The proper term for the letter we started with is the axiom, and each consecutive run we did with the find and replace rules are really called generations. This is the underlying principle of an L System. While this is cool, I'm sure you're asking how I turned the long string above into the animation at the beginning of this post.

Enter the turtle

For this next part you are going to need a (virtual) pet turtle. My turtle isn't just any turtle - he's a super fast racing turtle that I've trained to follow simple instructions like "go forwards", or "turn right". Now suppose I attach a pen to my turtle so that he leaves a line everywhere he walks.

Now if I give my turtle the following rules and the output from one of the generations above, I'll get back a picture:

f means go forwards one pace

h also mean go forwards one pace

+ means turn right

means turn left

ignore any other characters

Sierpinski triangle generation #4

Writing some code

Now that I've introduced how L Systems work and how you can use them to draw pretty pictures, we can start writing some code. I decided to use C♯ to simulate the L System, along with Mono.Cairo to draw the output. Mono.Cairo is a wonderful graphics drawing library that comes bundled along with the Mono runtime as standard.

I decided to split my implementation into 2 classes: the L System and the Turtle. Here's the code for the L System:

using System;
using System.Collections.Generic;
using System.IO;
using System.Runtime.InteropServices;
using System.Security.Policy;

namespace LSystem
{
    public class Rule
    {
        public string Find;
        public string Replace;

        public Rule(string inFind, string inReplace)
        {
            Find = inFind;
            Replace = inReplace;
        }
    }

    /// <summary>
    /// Simulates an L-System.
    /// Implemented according to http://www.cs.unm.edu/~joel/PaperFoldingFractal/L-system-rules.html
    /// </summary>
    public class LSystem
    {
        public readonly string Root;
        private List<Rule> rules = new List<Rule>();

        public int GenerationCount { get; private set; }
        public string CurrentGeneration { get; private set; }

        public Dictionary<string, string> Definitions { get; private set; }

        public LSystem(string inRoot)
        {
            CurrentGeneration = Root = inRoot;
            Definitions = new Dictionary<string, string>();
        }

        public void AddRule(string find, string replace)
        {
            rules.Add(new Rule(find, replace));
        }

        public string Simulate()
        {
            List<KeyValuePair<int, Rule>> rulePositions = new List<KeyValuePair<int, Rule>>();
            // Find all the current positions
            foreach(Rule rule in rules)
            {
                List<int> positions = AllIndexesOf(CurrentGeneration, rule.Find);
                foreach (int pos in positions)
                    rulePositions.Add(new KeyValuePair<int, Rule>(pos, rule));
            }
            rulePositions.Sort(compareRulePairs);

            string nextGeneration = CurrentGeneration;
            int replaceOffset = 0;
            foreach(KeyValuePair<int, Rule> rulePos in rulePositions)
            {
                int nextPos = rulePos.Key + replaceOffset;
                nextGeneration = nextGeneration.Substring(0, nextPos) + rulePos.Value.Replace + nextGeneration.Substring(nextPos + rulePos.Value.Find.Length);
                replaceOffset += rulePos.Value.Replace.Length - rulePos.Value.Find.Length;
            }
            CurrentGeneration = nextGeneration;
            GenerationCount++;
            return CurrentGeneration;
        }

        private int compareRulePairs(KeyValuePair<int, Rule> a, KeyValuePair<int, Rule> b)
        {
            return a.Key - b.Key;
        }

        /// <summary>
        /// From http://stackoverflow.com/a/2641383/1460422
        /// </summary>
        /// <param name="str"></param>
        /// <param name="value"></param>
        /// <returns></returns>
        private List<int> AllIndexesOf(string str, string value) {
            if (String.IsNullOrEmpty(value))
                throw new ArgumentException("the string to find may not be empty", "value");
            List<int> indexes = new List<int>();
            for (int index = 0;; index += value.Length) {
                index = str.IndexOf(value, index);
                if (index == -1)
                    return indexes;
                indexes.Add(index);
            }
        }

        public static LSystem FromFile(string filename)
        {
            StreamReader source = new StreamReader(filename);
            LSystem resultSystem = new LSystem(source.ReadLine());

            string nextLine = string.Empty;
            while (true) {
                nextLine = source.ReadLine();
                if (nextLine == null)
                    break;
                if (!nextLine.Contains("=") || nextLine.StartsWith("#") || nextLine.Trim().Length == 0)
                    continue;
                string[] parts = nextLine.Split(new char[]{'='}, 2);

                if(parts[0].StartsWith("!"))
                {
                    // This is a definition
                    resultSystem.Definitions.Add(parts[0].Trim('!'), parts[1]);
                }
                else
                {
                    resultSystem.AddRule(parts[0].Trim(), parts[1].Trim());
                }
            }
            return resultSystem;
        }
    }
}

There's quite a lot of code here, so I'll break it down. The most important bit is highlighted on lines 46-69. This Simulate() function takes the current generation and runs all the find and replace rules in parallel. A regular find and replace wouldn't work here because subsequent rules would pick up on new characters are were added when the previous rules added in a single generation.

Other important bits include the Rule class at the top, which represents a single find and replace rule, and the static FromFile() method. The FromFile() method loads a ruleset from a text file like this one, which produces a Dragon Curve:

f
f=f-h
h=f+h

....or this one, which produces a Sierpinski triangle, like the one above:

f
!angle=1.0472
h=f+h+f
f=h-f-h

The line beginning with an exclamation mark is a definition or a directive, which gives the turtle an instruction on how it should do something. They are stored in the string-to-string dictionary Definitions. The turtle class (which I'll show you in a moment) picks up on these and configures itself accordingly.

The turtle class I wrote is not as neat as the above. In fact it's kind of hacky. Here's what I ended up with:


using System;
using Cairo;
using System.Text.RegularExpressions;
using System.Resources;
using System.Collections.Generic;

namespace SimpleTurtle
{
    public class Area
    {
        public double X;
        public double Y;
        public double Width;
        public double Height;

        public Area(double inX, double inY, double inWidth, double inHeight)
        {
            X = inX;
            Y = inY;
            Width = inWidth;
            Height = inHeight;
        }
    }
    public class Turtle
    {
        private bool strictMode = false;
        private string commandQueue;

        private PointD position;
        private Area bounds;
        private double heading;
        private double headingStep;
        public double HeadingStep
        {
            get { return headingStep; }
            set { headingStep = value; }
        }
        private double movementStep ;
        public double MovementStep
        {
            get { return movementStep; }
            set { movementStep = value; }
        }

        public Turtle ()
        {
            Reset();
        }

        public void ApplyDefinitions(Dictionary<string, string> definitions)
        {
            foreach(KeyValuePair<string, string> definition in definitions)
            {
                switch(definition.Key.ToLower())
                {
                    case "angle":
                        HeadingStep = double.Parse(definition.Value);
                        break;
                }
            }
        }

        public bool Commands(string commandText)
        {
            // Remove all whitespace
            commandText = Regex.Replace(commandText, @"\s+", "");
            commandText = commandText.Replace("h", "f");

            string okCommands = "f+-";

            foreach(char ch in commandText)
            {
                if(okCommands.Contains(ch.ToString()))
                {
                    switch(ch)
                    {
                        case 'f':
                            Forwards();
                            break;
                        case '+':
                            Turn(false);
                            break;
                        case '-':
                            Turn(true);
                            break;
                        default:
                            if (strictMode) {
                                Console.WriteLine("The unexpected character '{0}' slipped through the net!", ch);
                                return false;
                            }
                            break;
                    }
                }
                else if(strictMode)
                {
                    Console.Error.WriteLine("Error: unexpected character '{0}'", ch);
                    return false;
                }
            }
            commandQueue += commandText;
            return true;
        }

        public void Draw(string filename, bool reset = true)
        {
            ImageSurface canvas = new ImageSurface(Format.ARGB32, (int)Math.Ceiling(bounds.Width + 10), (int)Math.Ceiling(bounds.Height + 10));
            Context context = new Context(canvas);
            PointD position = new PointD(-bounds.X + 5, -bounds.Y + 5);
            double heading = 0;

            context.LineWidth = 3;
            context.MoveTo(position);
            foreach(char ch in commandQueue)
            {
                switch(ch)
                {
                    case 'f':
                        PointD newPosition = new PointD(
                            position.X + movementStep * Math.Sin(heading),
                            position.Y + movementStep * Math.Cos(heading)
                        );
                        context.LineTo(newPosition);
                        position = newPosition;
                        break;
                    case '+':
                        heading += headingStep;
                        break;
                    case '-':
                        heading -= headingStep;
                        break;
                }
            }

            context.Stroke();
            canvas.WriteToPng(string.Format(filename));
            context.Dispose();
            canvas.Dispose();
            if(reset)
                Reset();
        }

        public void Forwards()
        {
            PointD newPosition = new PointD(
                position.X + MovementStep * Math.Sin(heading),
                position.Y + MovementStep * Math.Cos(heading)
            );

            if (newPosition.X > bounds.X + bounds.Width)
                bounds.Width += newPosition.X - position.X;
            if (newPosition.Y > bounds.Y + bounds.Height)
                bounds.Height += newPosition.Y - position.Y;
            if (newPosition.X < bounds.X)
            {
                bounds.X = newPosition.X;
                bounds.Width += position.X - newPosition.X;
            }
            if (newPosition.Y < bounds.Y)
            {
                bounds.Y = newPosition.Y;
                bounds.Height += position.Y - newPosition.Y;
            }

            position = newPosition;
        }

        public void Turn(bool anticlockwise = false)
        {
            if (!anticlockwise)
                heading += HeadingStep;
            else
                heading -= HeadingStep;
        }

        public void Reset()
        {
            commandQueue = string.Empty;
            position = new PointD(0, 0);
            bounds = new Area(position.X, position.Y, 1, 1);
            heading = 0;
            headingStep = Math.PI / 2;
            movementStep = 25;
        }
    }
}

Basically, the problem I found myself with was that I had to tell Cairo how large the canvas should be ahead of time, before I'd actually finished following all the commands that had been given. In order to get around this, I implemented the commands twice (bad practice, I know!).

The first implementation are the public interface methods, like Commands() (which processes a bunch of commands all at once), and the Forwards() and Turn() methods. The Commands() method follows a bunch of commands, using the private member variables at the top of the Turtle class to keep track of it's position. Each command it understands is then added to a store of valid commands ready for processing by the Draw() function (highlighted, lines 104-140). Whilst processing all these commands, the bounds of the area that the turtle has visited are tracked in the private bounds member variable (highlighted, line 30), which is of type Area (see the class at the top of the file).

The Draw() function creates a new Cairo canvas (lines 106-107) to the size of the bounds calculated previously (plus a bit extra of a border), and replays all the processed commands back on top of it. The final image is then dumped to disk.

After all that, we are missing one last piece of the puzzle: A bridge to connect the two together and provide an interface through which we can interact with the program. This is what I used:

using System;
using System.Collections.Generic;
using System.IO;

class Program
{
    public static void Main(string[] args)
    {
        if(args.Length < 2)
        {
            Console.WriteLine("Usage: ");
            Console.WriteLine(@"    ./Simpleturtle.exe <filename> <generations> [<stepCount> [<startingGeneration>]]");
            return;
        }

        int generations = int.Parse(args[1]);
        string filename = args[0];
        int movementStep = 25;
        if (args.Length > 2)
            movementStep = int.Parse(args[2]);
        int startingGeneration = 0;
        if (args.Length > 3)
            startingGeneration = int.Parse(args[3]);

        LSystem lsystem = LSystem.FromFile(filename);
        foreach(KeyValuePair<string, string> kvp in lsystem.Definitions)
        {
            Console.WriteLine("Setting {0} to {1}.", kvp.Key, kvp.Value);
        }
        Turtle turtle = new Turtle();

        for(int i = startingGeneration; i < generations; i++)
        {
            turtle.ApplyDefinitions(lsystem.Definitions);
            turtle.MovementStep = movementStep;
            turtle.Commands(lsystem.CurrentGeneration);
            turtle.Draw(string.Format("lsystem_generation_{0}.png", i));
            lsystem.Simulate();
            File.WriteAllText(string.Format("lsystem_generation_{0}.txt", i), lsystem.CurrentGeneration);
            Console.WriteLine("Generation {0} completed.", i);
        }
    }
}

There's nothing too interesting here, just a help message and a parameter reading code. If you put all of this code together, though, you can generate pretty pictures like the ones I showed you above.

This is only the beginning of what you can do with L Systems though. You could extend it to work in 3D, or give your turtle teleporting powers (look half way down) and draw some trees. you could even edit a few characters randomly after completing the main simulation to make every result slightly different. The possibilities are endless!

Sources and Further Reading

Tips on writing annoying / difficult code

If you've been programming for any length of time, you'll know that eventually you will have to implement a particularly complex algorithm or conquer some other especially difficult challenge. Maybe it's implementing specular lighting, or perhaps it's fixing a nasty annoying bug in your physics engine.

Since I seem to have been doing this with increasing frequency at the moment due to University coursework, I thought I'd share a few things with you that I personally find helpful.

Plan out what you intend to do. Create a flowchart. Fill a whiteboard with notes. Talk to someone. Even if it's a rubber duck. Planning out what you want to do ensures that you have a clear idea in your head as to what you want to accomplish and how to go about doing it. It's very easy to fall into the trap of doing random things and just hoping that it will work.

Break the task down into steps. This goes hand in hand with the above point. Break your task down into small, manageable steps that you can follow one at a time to achieve your goal. I often find if I don't do this the problem I'm trying to solve looks an awful lot bigger than it really is.

Tackle the steps one at a time, and test regularly. Skipping to step 3 when you've barely started step 1 might be tempting, but it will probably cause more problems that it solves and lead to confusion. If you're thinking of doing this, then you might not have your steps in the right order.

Similarly, doing steps 1-4 before testing what you've done also isn't a good idea, because you might have made a mistake in step 1 that affects the output of step 4, and then spend an hour debugging step 4's code only to realise it is fine and it's actually the code you wrote in the beginning that is causing the problem.

I hope you find these tips helpful. If you did, leave a comment below!

Capturing and sending error reports by email in C♯

A month or two ago I put together a simple automatic updater and showed you how to do the same. This time I'm going to walk you through how to write your own error reporting system. It will be able to catch any errors through a try...catch block, ask the user whether they want to send an error report when an exception is thrown, and send the report to your email inbox.

Just like last time, this post is a starting point, not an ending point. It has a significant flaw and can be easily extended to, say, ask the user to write a short paragraph detailing what they were doing at the time of the crash, or add a proper gui, for example.

Please note that this tutorial requires that you have a server of some description to use to send the error reports to. If you want to get the system to send you an email too, you'll need a working mail server. Thankfully DigitalOcean provide free credit if you have the GitHub Student pack. This tutorial assumes that your mail server (or at least a relay one) is running on the same machine as your web server. While setting one up correctly can be a challenge, Lee Hutchinson over at Ars Technica has a great tutorial that's easy to follow.

To start with, we will need a suitable test program to work with whilst building this thing. Here's a good one riddled with holes that should throw more than a few exceptions:

using System;
using System.IO;
using System.Net;
using System.Text;

public class Program
{
    public static readonly string Name = "Dividing program";
    public static readonly string Version = "0.1";

    public static string ProgramId
    {
        get { return string.Format("{0}/{1}", Name, Version); }
    }

    public static int Main(string[] args)
    {
        float a = 0, b = 0, c = 0;

        Console.WriteLine(ProgramId);
        Console.WriteLine("This program divides one number by another.");

        Console.Write("Enter number 1: ");
        a = float.Parse(Console.ReadLine());
        Console.Write("Enter number 2: ");
        b = float.Parse(Console.ReadLine());

        c = a / b;
        Console.WriteLine("Number 1 divided by number 2 is {0}.", c);

        return 0;
}

There are a few redundant using statements at the top there - we will get to utilizing them later on.

First things first - we need to capture all exceptions and build an error report:

try
{
    // Insert your program here
}
catch(Exception error)
{
    Console.Write("Collecting data - ");
    MemoryStream dataStream  = new MemoryStream();
    StreamWriter dataIn = new StreamWriter(dataStream);
    dataIn.WriteLine("***** Error Report *****");
    dataIn.WriteLine(error.ToString());
    dataIn.WriteLine();
    dataIn.WriteLine("*** Details ***");
    dataIn.WriteLine("a: {0}", a);
    dataIn.WriteLine("b: {0}", b);
    dataIn.WriteLine("c: {0}", c);
    dataIn.Flush();

    dataStream.Seek(0, SeekOrigin.Begin);
    string errorReport = new StreamReader(dataStream).ReadToEnd();
    Console.WriteLine("done");
}

If you were doing this for real, it might be a good idea to move all of your application logic it it's own class and have a call like application.Run() instead of placing your code directly inside the try{ } block. Anyway, the above will catch the exception, and build a simple error report. I'm including the values of a few variables I created too. You might want to set up your own mechanism for storing state data so that the error reporting system can access it, like a special static class or something.

Now that we have created an error report, we need to send it to the server to processing. Before we do this, though, we ought to ask the user if this is ok with them (it is their computer in all likeliness after all!). This is easy:

Console.WriteLine("An error has occurred!");
Console.Write("Would you like to report it? [Y/n] ");

bool sendReport = false;
while(true)
{
    ConsoleKey key = Console.ReadKey().Key;
    if (key == ConsoleKey.Y) {
        sendReport = true;
        break;
    }
    else if (key == ConsoleKey.N)
        break;
}
Console.WriteLine();

if(!sendReport)
{
    Console.WriteLine("No report has been sent.");
    Console.WriteLine("Press any key to exit.");
    Console.ReadKey(true);
    return 1;
}

Since this program uses the console, I'm continuing that trend here. You will need to create your own GUI if you aren't creating a console app.

Now that's taken care of, we can go ahead and send the report to the server. Here's how I've done it:

Console.Write("Sending report - ");
HttpWebRequest reportSender = WebRequest.CreateHttp("https://starbeamrainbowlabs.com/reportSender.php");
reportSender.Method = "POST";
byte[] payload = Encoding.UTF8.GetBytes(errorReport);
reportSender.ContentType = "text/plain";
reportSender.ContentLength = payload.Length;
reportSender.UserAgent = ProgramId;
Stream requestStream = reportSender.GetRequestStream();
requestStream.Write(payload, 0, payload.Length);
requestStream.Close();

WebResponse reportResponse = reportSender.GetResponse();
Console.WriteLine("done");
Console.WriteLine("Server response: {0}", ((HttpWebResponse)reportResponse).StatusDescription);
Console.WriteLine("Press any key to exit.");
Console.ReadKey(true);
return 1;

That may look unfamiliar and complicated, so let's walk through it one step at a time.

To start with, I create a new HTTP web request and point it at an address on my server. You will use a slightly different address, but the basic principle is the same. As for what resides at that address - we will take a look at that later on.

Next I set request method to be POST so that I can send some data to the server, and set a few headers to help the server out in understanding our request. Then I prepare the error report for transport and push it down the web request's request stream.

After that I get the response from the server and tell the user that we have finished sending the error report to the server.

That pretty much completes the client side code. Here's the whole thing from start to finish:

using System;
using System.IO;
using System.Net;
using System.Text;

public class Program
{
    public static readonly string Name = "Dividing program";
    public static readonly string Version = "0.1";

    public static string ProgramId
    {
        get { return string.Format("{0}/{1}", Name, Version); }
    }

    public static int Main(string[] args)
    {
        float a = 0, b = 0, c = 0;

        try
        {
            Console.WriteLine(ProgramId);
            Console.WriteLine("This program divides one number by another.");

            Console.Write("Enter number 1: ");
            a = float.Parse(Console.ReadLine());
            Console.Write("Enter number 2: ");
            b = float.Parse(Console.ReadLine());

            c = a / b;
            Console.WriteLine("Number 1 divided by number 2 is {0}.", c);
        }
        catch(Exception error)
        {
            Console.WriteLine("An error has occurred!");
            Console.Write("Would you like to report it? [Y/n] ");

            bool sendReport = false;
            while(true)
            {
                ConsoleKey key = Console.ReadKey().Key;
                if (key == ConsoleKey.Y) {
                    sendReport = true;
                    break;
                }
                else if (key == ConsoleKey.N)
                    break;
            }
            Console.WriteLine();

            if(!sendReport)
            {
                Console.WriteLine("No report has been sent.");
                Console.WriteLine("Press any key to exit.");
                Console.ReadKey(true);
                return 1;
            }

            Console.Write("Collecting data - ");
            MemoryStream dataStream  = new MemoryStream();
            StreamWriter dataIn = new StreamWriter(dataStream);
            dataIn.WriteLine("***** Error Report *****");
            dataIn.WriteLine(error.ToString());
            dataIn.WriteLine();
            dataIn.WriteLine("*** Details ***");
            dataIn.WriteLine("a: {0}", a);
            dataIn.WriteLine("b: {0}", b);
            dataIn.WriteLine("c: {0}", c);
            dataIn.Flush();

            dataStream.Seek(0, SeekOrigin.Begin);
            string errorReport = new StreamReader(dataStream).ReadToEnd();
            Console.WriteLine("done");

            Console.Write("Sending report - ");
            HttpWebRequest reportSender = WebRequest.CreateHttp("https://starbeamrainbowlabs.com/reportSender.php");
            reportSender.Method = "POST";
            byte[] payload = Encoding.UTF8.GetBytes(errorReport);
            reportSender.ContentType = "text/plain";
            reportSender.ContentLength = payload.Length;
            reportSender.UserAgent = ProgramId;
            Stream requestStream = reportSender.GetRequestStream();
            requestStream.Write(payload, 0, payload.Length);
            requestStream.Close();

            WebResponse reportResponse = reportSender.GetResponse();
            Console.WriteLine("done");
            Console.WriteLine("Server response: {0}", ((HttpWebResponse)reportResponse).StatusDescription);
            Console.WriteLine("Press any key to exit.");
            Console.ReadKey(true);
            return 1;
        }

        return 0;
    }
}

(Pastebin, Raw)

Next up is the server side code. Since I'm familiar with it and it can be found on all popular web servers, I'm going to be using PHP here. You could write this in ASP.NET, too, but I'm not familiar with it, nor do I have the appropriate environment set up at the time of posting (though I certainly plan on looking into it).

The server code can be split up into 3 sections: the settings, receiving and extending the error report, and sending the error report on in an email. Part one is quite straightforward:

<?php
/// Settings ///
$settings = new stdClass();
$settings->fromAddress = "postasaurus@starbeamrainbowlabs.com";
$settings->toAddress = "bugs@starbeamrainbowlabs.com";

The above simply creates a new object and stores a few settings in it. I like to put settings at the top of small scripts like this because it both makes it easy to reconfigure them and allows for expansion later.

Next we need to receive the error report from the client:

// Get the error report from the client
$errorReport = file_get_contents("php://input");

PHP on a web server it smarter than you'd think and collects some useful information about the connected client, so we can collect a few interesting statistics and tag them onto the end of the error report like this:

// Add some extra information to it
$errorReport .= "\n*** Server Information ***\n";
$errorReport .= "Date / time reported: " . date("r") . "\n";
$errorReport .= "Reporting ip: " . $_SERVER['REMOTE_ADDR'] . "\n";
if(isset($_SERVER["HTTP_X_FORWARDED_FOR"]))
{
    $errorReport .= "The error report was forwarded through a proxy.\n";
    $errorReport .= "The proxy says that it forwarded the request from this address: " . $_SERVER['HTTP_X_FORWARDED_FOR'] . "\n\n";
}
if(isset($_SERVER["HTTP_USER_AGENT"]))
{
    $errorReport .= "The reporting client identifies themselves as: " . $_SERVER["HTTP_USER_AGENT"] . ".\n";
}

I'm adding the date and time here too just because the client could potentially fake it (they could fake everything, but that's a story for another time). I'm also collecting the client's user agent string too. This is being set in the client code above to the name and version of the program running. This information could be useful if you attach multiple programs to the same error reporting script. You could modify the client code to include the current .NET version, too by utilising Environment.Version.

Lastly, since the report has gotten this far, we really should do something with it. I decided I wanted to send it to myself in an email, but you could just as easily store it in a file using something like file_put_contents("bug_reports.txt", $errorReport, FILE_APPEND);. Here's the code I came up with:

$emailHeaders = [
    "From: $settings->fromAddress",
    "Content-Type: text/plain",
    "X-Mailer: PHP/" . phpversion()
];

$subject = "Error Report";
if(isset($_SERVER["HTTP_USER_AGENT"]))
    $subject .= " from " . $_SERVER["HTTP_USER_AGENT"];

mail($settings->toAddress, $subject, $errorReport, implode("\r\n", $emailHeaders), "-t");

?>

That completes the server side code. Here's the completed script:

<?php
/// Settings ///
$settings = new stdClass();
$settings->fromAddress = "postasaurus@starbeamrainbowlabs.com";
$settings->toAddress = "bugs@starbeamrainbowlabs.com";

// Get the error report from the client
$errorReport = file_get_contents("php://input");

// Add some extra information to it
$errorReport .= "\n*** Server Information ***\n";
$errorReport .= "Date / time reported: " . date("r") . "\n";
$errorReport .= "Reporting ip: " . $_SERVER['REMOTE_ADDR'] . "\n";
if(isset($_SERVER["HTTP_X_FORWARDED_FOR"]))
{
    $errorReport .= "The error report was forwarded through a proxy.\n";
    $errorReport .= "The proxy says that it forwarded the request from this address: " . $_SERVER['HTTP_X_FORWARDED_FOR'] . "\n\n";
}
if(isset($_SERVER["HTTP_USER_AGENT"]))
{
    $errorReport .= "The reporting client identifies themselves as: " . $_SERVER["HTTP_USER_AGENT"] . ".\n";
}

$emailHeaders = [
    "From: $settings->fromAddress",
    "Content-Type: text/plain",
    "X-Mailer: PHP/" . phpversion()
];

$subject = "Error Report";
if(isset($_SERVER["HTTP_USER_AGENT"]))
    $subject .= " from " . $_SERVER["HTTP_USER_AGENT"];

mail($settings->toAddress, $subject, $errorReport, implode("\r\n", $emailHeaders), "-t");

?>

(Pastebin, Raw)

The last job we need to do is to upload the PHP script to a PHP-enabled web server, and go back to the client and point it at the web address at which the PHP script is living.

If you have read this far, then you've done it! You should have by this point a simple working error reporting system. Here's an example error report email that I got whilst testing it:

***** Error Report *****
System.FormatException: Input string was not in a correct format.
  at System.Number.ParseSingle (System.String value, NumberStyles options, System.Globalization.NumberFormatInfo numfmt) <0x7fe1c97de6c0 + 0x00158> in <filename unknown>:0 
  at System.Single.Parse (System.String s, NumberStyles style, System.Globalization.NumberFormatInfo info) <0x7fe1c9858690 + 0x00016> in <filename unknown>:0 
  at System.Single.Parse (System.String s) <0x7fe1c9858590 + 0x0001d> in <filename unknown>:0 
  at Program.Main (System.String[] args) <0x407d7d60 + 0x00180> in <filename unknown>:0 

*** Details ***
a: 4
b: 0
c: 0

*** Server Information ***
Date / time reported: Mon, 11 Apr 2016 10:31:20 +0100
Reporting ip: 83.100.151.189
The reporting client identifies themselves as: Dividing program/0.1.

I mentioned at the beginning of this post that that this approach has a flaw. The main problem lies in the fact that the PHP script can be abused by a knowledgeable attacker to send you lots of spam. I can't think of any real way to properly solve this, but I'd suggest storing the PHP script at a long and complicated URL that can't be easily guessed. There are probably other flaws as well, but I can't think of any at the moment.

Found a mistake? Got an improvement? Please leave a comment below!

Test your next web based idea with RawGit

All too often I find myself wanting to test something web based quickly without starting a local web server (this is how my procedural castle generator started for example).

While a quick Github Gist or a Github Repository would be a great place to host things like this Github serves your code with the mime type text/plain, and the X-Content-Type-Options: nosniff header. Without the extra header your browser will inspect the content that is being sent by the server and automatically determine the actual content type - but the above header disables this functionality, rendering both Github Gist and a Github Repository useless for anything other than sharing code.

Thankfully all is not lost. Some time ago ago I discovered RawGit. RawGit is a service that lets you enter the raw view url of either a Github gist or a file in a Github repository, and it will serve it with the correct content type, allowing you to use gist and Github repositories to house your latest and greatest experiment.

It has a different url for development and production use, too. The development url doesn't cache anything and updates instantly, whereas the production url is cached indefinitely and is delivered via a CDN (content delivery network).

Because different files in the same folder or gist generate a similar url, you can include them in your HTML page in the same way as if you were doing it locally:

<!DOCTYPE html>
<html>
    <head>
        <meta charset='utf-8' />
        <title>Test page</title>
    </head>
    <body>
        <p>Page content here</p>

        <!---------------->
        <script src="tomatoes.js"></script>
    </head>
</html>

Pepperminty Wiki is now on WikiMatrix!

The WikiMatrix Logo

After a year of waiting (and hard work improving it), Pepperminty Wiki is now on WikiMatrix.org!

WikiMatrix is a brilliant site that lets you compare many different pieces of wiki software with each other in order to figure out which one best meets your needs. Unfortunately, the admins are rather difficult to get hold of, and so new wikis get added rarely.

If you're looking to set up a a small wiki, I recommend checking out my project Pepperminty Wiki. It's inspired by am2064's logo am2064's Minty Wiki that he posted on /r/tinycode, and is designed to be uploaded to a web server and just work (after configuring it, of course!). It's packed into a single file and currently weighs in at ~160kb, so you won't have to wait around for ages for it to download or unpack.

Notable features include multiple users, subpages, templates, file uploads, a full text search engine, page protection, and more! New features are being added all the time - an up to date list can be found in the Github README (link below). Features can be added or removed at any time, too - Pepperminty Wiki sports a module based approach, letting you decide on what features your wiki has enabled.

Link: Pepperminty Wiki

Pepperminty Wiki

Happy Easter 2016!

Hello!

This is just a short post to wish you a very Happy Easter!

I don't have anything to show you this easter (mainly because University ACWs have been taking up all my time :() - hopefully I'll get around to building something for next easter :)

Art by Mythdael