Starbeamrainbowlabs

Stardust
Blog


Archive


Mailing List Articles Atom Feed Comments Atom Feed Twitter Reddit Facebook

Tag Cloud

3d 3d printing account algorithms android announcement architecture archives arduino artificial intelligence artix assembly async audio automation backups bash batch blog bookmarklet booting bug hunting c sharp c++ challenge chrome os code codepen coding conundrums coding conundrums evolved command line compilers compiling compression css dailyprogrammer data analysis debugging demystification distributed computing documentation downtime electronics email embedded systems encryption es6 features ethics event experiment external first impressions future game github github gist gitlab graphics hardware hardware meetup holiday holidays html html5 html5 canvas infrastructure interfaces internet interoperability io.js jabber jam javascript js bin labs learning library linux lora low level lua maintenance manjaro network networking nibriboard node.js operating systems own your code performance phd photos php pixelbot portable privacy problem solving programming problems project projects prolog protocol protocols pseudo 3d python reddit redis reference releases resource review rust searching secrets security series list server software sorting source code control statistics storage svg talks technical terminal textures thoughts three thing game three.js tool tutorial twitter ubuntu university update updates upgrade version control virtual reality virtualisation visual web website windows windows 10 xmpp xslt

Summer Project Part 5: When is a function not a function?

Another post! Looks like I'm on a roll in this series :P

In the last post, I looked at the box I designed that was ready for 3D printing. That process has now been completed, and I'm now in possession of an (almost) luminous orange and pink box that could almost glow in the dark.......

I also looked at the libraries that I'll be using and how to manage the (rather limited) amount of memory available in the AVR microprocessor.

Since last time, I've somehow managed to shave a further 6% program space off (though I'm not sure how I've done it), so most recently I've been implementing 2 additional features:

  • An additional layer of AES encryption, to prevent The Things Network for having access to the decrypted data
  • GPS delta checking (as I'm calling it), to avoid sending multiple messages when the device hasn't moved

After all was said and done, I'm now at 97% program space and 47% global variable RAM usage.

To implement the additional AES encryption layer, I abused LMiC's IDEETRON AES-128 (ECB mode) implementation, which is stored in src/aes/ideetron/AES-128_V10.cpp.

It's worth noting here that if you're doing crypto yourself, it's seriously not recommended that you use ECB mode. Please don't. The only reason that I used it here is because I already had an implementation to hand that was being compiled into my program, I didn't have the program space to add another one, and my messages all start with a random 32-bit unsigned integer that will provide a measure of protection against collision attacks and other nastiness.

Specifically, it's the method with this signature:

void lmic_aes_encrypt(unsigned char *Data, unsigned char *Key);

Since this is an internal LMiC function declared in a .cpp source file with no obvious header file twin, I needed to declare the prototype in my source code as above - as the method will only be discovered by the compiler when linking the object files together (see this page for more information about the C++ compilation process. While it's for regular Linux executable binaries, it still applies here since the Arduino toolchain spits out a very similar binary that's uploaded to the microprocessor via a programmer).

However, once I'd sorted out all the typing issues, I slammed into this error:

/tmp/ccOLIbBm.ltrans0.ltrans.o: In function `transmit_send':
sketch/transmission.cpp:89: undefined reference to `lmic_aes_encrypt(unsigned char*, unsigned char*)'
collect2: error: ld returned 1 exit status

Very strange. What's going on here? I declared that method via a prototype, didn't I?

Of course, it's not quite that simple. The thing is, the file I mentioned above isn't the first place that a prototype for that method is defined in LMiC. It's actually in other.c, line 35 as a C function. Since C and C++ (for all their similarities) are decidedly different, apparently to call a C function in C++ code you need to declare the function prototype as extern "C", like this:

extern "C" void lmic_aes_encrypt(unsigned char *Data, unsigned char *Key);

This cleaned the error right up. Turns out that even if a function body is defined in C++, what matters is where the original prototype is declared.

I'm hoping to release the source code, but I need to have a discussion with my supervisor about that at the end of the project.

Found this interesting? Come across some equally nasty bugs? Comment below!

Summer Project Part 4: Threading the needle and compacting it down

In the last part, I put the circuit for the IoT device together and designed a box for said circuit to be housed inside of.

In this post, I'm going to talk a little bit about 3D printing, but I'm mostly going to discuss the software aspect of the firmware I'm writing for the Arduino Uno that's going to be control the whole operation out in the field.

Since last time, I've completed the design for the housing and sent it off to my University for 3D printing. They had some great suggestions for improving the design like making the walls slightly thicker (moving from 2mm to 4mm), and including an extra lip on the lid to keep it from shifting around. Here are some pictures:

(Left: The housing itself. Right: The lid. On the opposite side (not shown), the screw holes are indented.)

At the same time as handling sending the housing off to be 3D printed, I've also been busily iterating on the software that the Arduino will be running - and this is what I'd like to spend the majority of this post talking about.

I've been taking an iterative approach to writing it - adding a library, interfacing with it and getting it to do what I want on it's own, then integrating it into the main program.... and then compacting the whole thing down so that it'll fit inside the Arduino Uno. The thing is, the Uno is powered by an ATmega328P (datasheet). Which has 32K of program space and just 2K of RAM. Not much. At all.

The codebase I've built for the Uno is based on the following libraries.

  • LMiC (the matthijskooijman fork) - the (rather heavy and needlessly complicated) LoRaWAN implementation
  • Entropy for generating random numbers as explained in part 2
  • TinyGPS, for decoding NMEA messages from the NEO-6M
  • SdFat, for interfacing with microSD cards over SPI

Memory Management

Packing the whole program into a 32K + 2K box is not exactly an easy challenge, I discovered. I chose to first deal with the RAM issue. This was greatly aided by the FreeMemory library, which tells you how much RAM you've got left at a given point in the execution of your program. While it's a bit outdated, it's still a useful tool. It works a bit like this:

#include <MemoryFree.h>;

void setup() {
    Serial.begin(115200);
    Serial.println(freeMemory, DEC);
    char test[] = "Bobs Rockets";
    Serial.println(freeMemory, DEC); // Should be lower than the above call
}

void loop() {
    // Nothing here
}

It's worth taking a moment to revise the way stacks and heaps work - and the differences between how they work in the Arduino environment and on your desktop. This is going to get rather complicated quite quickly - so I'd advise reading this stack overflow answer first before continuing.

First, let's look at the locations in RAM for different types of allocation:

  • Things on the stack
  • Things on the heap
  • Global variables

Unlike on the device you're reading this on, the Arduino does not support multiple processes - and therefore the entirety of the RAM available is allocated to your program.

Since I wasn't sure about preecisly how the Arduino does it (it's processor architecture-specific), I wrote a simple test program to tell me:

#include <Arduino.h>

struct Test {
    uint32_t a;
    char b;
};

Test global_var;

void setup() {
    Serial.begin(115200);

    Test stack;
    Test* heap = new Test();

    Serial.print(F("Stack location: "));
    Serial.println((uint32_t)(&stack), DEC);

    Serial.print(F("Heap location: "));
    Serial.println((uint32_t)heap, DEC);

    Serial.print(F("Global location: "));
    Serial.println((uint32_t)&global_var, DEC);
}

void loop() {
    // Nothing here
}

This prints the following for me:

Stack location: 2295
Heap location: 461
Global location: 284

From this we can deduce that global variables are located at the beginning of the RAM space, heap allocations go on top of globals, and the stack grows down starting from the end of RAM space. It's best explained with a diagram:

Now for the differences. On a normal machine running an operating system, there's an extra layer of abstraction between where things are actually located in RAM and where the operating system tells you they are located. This is known as virtual memory address translation (see also virtual memory, virtual address space).

It's a system whereby the operating system maintains a series of tables that map physical RAM to a virtual address space that the running processes actually use. Usually each process running on a system will have it's own table (but this doesn't mean that it will have it's own physical memory - see also shared memory, but this is a topic for another time). When a process accesses an area of memory with a virtual address, the operating system will transparently translate the address using the table to the actual location in RAM (or elsewhere) that the process wants to access.

This is important (and not only for security), because under normal operation a process will probably allocate and deallocate a bunch of different lumps of memory at different times. With a virtual address space, the operating system can defragment the physical RAM space in the background and move stuff around without disturbing currently running processes. Keeping the free memory contiguous speeds up future allocations, and ensures that if a process asks for a large block of contiguous memory the operating system will be able to allocate it without issue.

As I mentioned before though, the Arduino doesn't have a virtual memory system - partly because it doesn't support multiple processes (it would need an operating system for that). The side-effect here is that it doesn't defragment the physical RAM. Since C/C++ isn't a managed language, we don't get _heap compaction_ either like in .NET environments such as Mono.

All this leads us to an environment in which heap allocation needs to be done very carefully, in order to avoid fragmenting the heap and causing a stack crash. If an object somewhere in the middle of the heap is deallocated, the heap will not shrink until everything to the right of it is also deallocated. This post has a good explanation of the problem too.

Other things we need to consider are keeping global variables to a minimum, and trying to keep most things on the stack if we can help it (though this may slow the program down if it's copying things between stack frames all the time).

To this end, we need to choose the libraries we use with care - because they can easily break these guidelines that we've set for ourselves. For example, the inbuilt SD library is out, because it uses a global variable that eats over 50% of our available RAM - and it there's no way (that I can see at least) to reclaim that RAM once we're finished with it.

This is why I chose SdFat instead, because it's at least a little better at allowing us to reclaim most of the RAM it used once we're finished with it by letting the instance fall out of scope (though in my testing I never managed to reclaim all of the RAM it used afterwards).

Alternatives like µ-Fat do exist and are even lighter, but they have restrictions such as no appending to files for example - which would make the whole thing much more complicated since we'd have to pre-allocate the space for the file (which would get rather messy).

The other major tactic you can do to save RAM is to use the F() trick. Consider the following sketch:

#include 

void setup() {
    Serial.begin(115200);
    Serial.println("Bills boosters controller, version 1");
}

void loop() {
    // Nothing here
}

On the highlighted line we've got an innocent Serial.println() call. What's not obvious here is that the string literal here is actually copied to RAM before being passed to Serial.println() - using up a huge amount of our precious working memory. Wrapping it in the F() macro forces it to stay in your program's storage space:

Serial.println(F("Bills boosters controller, version 1"));

Saving storage

With the RAM issue mostly dealt with, I then had to deal with the thorny issue of program space. Unfortunately, this is not as easy as saving RAM because we can't just 'unload' something when it's not needed.

My approach to reducing program storage space was twofold:

  • Picking lightweight alternatives to libraries I needed
  • Messing with flags of said libraries to avoid compiling parts of libraries I don't need

It is for these reasons that I ultimately went with TinyGPS instead of TinyGPS++, as it saved 1% or so of the program storage space.

It's also for this reason that I've disabled as much of LMiC as possible:

#define DISABLE_JOIN
#define DISABLE_PING
#define DISABLE_BEACONS
#define DISABLE_MCMD_DCAP_REQ
#define DISABLE_MCMD_DN2P_SET

This disables OTAA, Class B functionality (which I don't need anyway), receiving messaages, the duty cycle cap system (which I'm not sure works between reboots), and a bunch of other stuff that I'd probably find rather useful.

In the future, I'll probably dedicate an entire microcontroller to handling LoRaWAN functionality - so that I can still use the features I've had to disable here.

Even doing all this, I still had to trim down my Serial.println() calls and remove any non-essential code to bring it under the 32K limit. As of the time of typing, I've got jut 26 bytes to spare!

Next time, after tuning the TPL5110 down to the right value, we're probably going to switch gears and look at the server-side of things - and how I'm going to be storing the data I receive from the Arudino-based device I've built.

Found this interesting? Got a suggestion? Comment below!

Summer Project Part 3: Putting it together

In the first post in this series, I outlined my plans for my Msc summer project and what I'm going to be doing. In the second post, I talked about random number generation for the data collection.

In this post, I'm going to give a general progress update - which will mostly centre around the Internet of Things device I'm building to collect the signal strength data.

Since the last post, I've got nearly all the parts I need for the project, except the TPL5111 power manager and 4 rechargeable AA batteries (which should be easy to come by - I'm sure I've got some lying around somewhere).

I've also wired the thing up, with a cable standing in for the TPL5111.

The IoT device all wired up. It basically consists of an Arduino Uno with a red Dragino LoRa shield on top, with a pair of small breadboards containing the peripherals and black power management boards respectively.

The power management board there technically doesn't need a breadboard, but it makes mounting it in the box easier.

I still need to splice the connector onto the battery box I had lying around with some soldering and electrical tape - I'll do that later this week.

The wiring there is kind of messy, but I've tested each device individually and they all appear to work as intended. Here's a clearer diagram of what's going on that drew up in Fritzing (sudo apt install fritzing for Linux users):

Speaking of mounting things in the box, I've discovered OpenSCAD thanks to help from a friend and have been busily working away at designing a box to put everything in that can be 3D printed:

I've just got the lid to do next (which I'm going to do after writing this blog post), and then I'm going to get it printed.

With this all done, it's time to start working on the transport for the messages - namely using LMIC to connection to the network and send the GPS location to the application server, which is also unfinished.

The lovely people at the hardware meetup have lent me a full 8-channel LoRaWAN gateway that's connected to The Things Network for my project, which will make this process a lot easier.

Next time, I'll likely talk about 3D printing and how I've been 'threading the needle', so to speak.

Summer Project Part 2: Random Number Analysis with Gnuplot

In my last post about my Masters Summer Project, I talked about my plans and what I'm doing. In this post, I want to talk about random number generator evaluation.

As part of the Arduino-based Internet of Things device that will be collecting the data, I need to generate high-quality random numbers in order to ensure that the unique ids I use in my project are both unpredictable and unique.

In order to generate such numbers, I've found a library that exploits the jitter in the inbuilt watchdog timer that's present in the Arduino Uno. It's got a manual which is worth a read, as it explains the concepts behind it quite well.

After some experimenting, I ended up with a program that would generate random numbers as fast as it could:

// Generate_Random_Numbers - This sketch makes use of the Entropy library
// to produce a sequence of random integers and floating point values.
// to demonstrate the use of the entropy library
//
// Copyright 2012 by Walter Anderson
//
// This file is part of Entropy, an Arduino library.
// Entropy is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// Entropy is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with Entropy.  If not, see <http://www.gnu.org/licenses/>.
// 
// Edited by Starbeamrainbowlabs 2019

#include "Entropy.h"

void setup() {
    Serial.begin(9600);

    // This routine sets up the watch dog timer with interrupt handler to maintain a
    // pool of real entropy for use in sketches. This mechanism is relatively slow
    // since it will only produce a little less than two 32-bit random values per
    // second.
    Entropy.initialize();
}

void loop() {
    uint32_t random_long;
    random_long = Entropy.random();
    Serial.println(random_long);
}

As you can tell, it's based on one of the examples. You may need to fiddle around with the imports in order to get it to work, because the Arduino IDE is terrible.

With this in place and uploaded to an Arduino, all I needed to do was log the serial console output to a file. Thankfully, this is actually really quite easy on Linux:

screen -S random-number-generator dd if=/dev/ttyACM0 of=random.txt bs=1

Since I connected the Arduino in question to a Raspberry Pi I have acting as a file server, I've included a screen call here that ensures that I can close the SSH session without it killing the command I'm executing - retaining the ability to 'reattach' to it later to check on it.

With it set off, I left it for a day or 2 until I had at least 1 MiB of random numbers. Once it was done, I ended up with a file looking a little like this:

216767155
986748290
455286059
1956258942
4245729381
3339111661
1821899502
3892736709
3658303796
2524261768
732282824
999812729
1312753534
2810553575
246363223
4106522438
260211625
1375011617
795481000
319056836

(Want more? Download the entire set here.)

In total, it generated 134318 numbers for me to play with, which should be plenty to graph their distribution.

Graphing such a large amount of numbers requires a special kind of program. Since I've used it before, I reached for Gnuplot.

A histogram is probably the best kind of graph for this purpose, so I looked up how to get gnuplot to draw one and tweaked it for my own purposes.

I didn't realise that you could do arbitrary calculations inside a Gnuplot graph definition file, but apparently you can. The important bit below is the bin_width variable:

set key off
set border 3

# Add a vertical dotted line at x=0 to show centre (mean) of distribution.
#set yzeroaxis

# Each bar is half the (visual) width of its x-range.
#set boxwidth 0.05 absolute
#set style fill solid 1.0 noborder

bin_width = 171797751.16;

bin_number(x) = floor(x / bin_width)

rounded(x) = bin_width * ( bin_number(x) + 0.5 )

set terminal png linewidth 3 size 1920,1080

plot 'random.txt' using (rounded($1)):(1) smooth frequency with boxes

It specifies the width of each bar on the graph. To work this out, we need to know the maximum number in the dataset. Then we can divide it by the target number of bins to get the width thereof. Time for some awk!

awk 'BEGIN { a = 0; b=999999999999999 } { if ($1>0+a) a=$1; if ($1 < 0+b) b=$1; } END { print(a, b); }' <random.txt

This looks like a bit of a mess, so let's unwind that awk script so that we can take a better look at it.

BEGIN {
    max = 0;
    min = 999999999999999
}
{
    if ($1 > max)
        max = $1;
    if ($1 < min)
        min = $1;
}
END {
    print("Max:", max, "Min:", min);
}

Much better. In short, it keeps a record of the maximum and minimum numbers it's seen so far, and updates them if it sees a better one. Let's run that over our random numbers:

awk -f minmax.awk 

Excellent. 4294943779 ÷ 25 = 171797751.16 - which is how I arrived at that value for the bin_width earlier.

Now we can render our graph:

gnuplot random.plt >histogram.png && optipng -o7 histogram.png

I always optimise images with either optipng or jpegoptim to save on storage space on the server, and bandwidth for readers - and in this case the difference was particularly noticeable. Here's the final graph:

The histograph generated by the above command.

As you can see, the number of numbers in each bin is pretty even, so we can reasonably conclude that the algorithm isn't too terrible.

What about uniqueness? Well, that's much easier to test than the distribution. If we count the numbers before and after removing duplicates, it should tell us how many duplicates there were. There's even a special command for it:

wc -l <random.txt 
134318
sort <random.txt | uniq | wc -l
134317
sort <random.txt | uniq --repeated
1349455381

Very interesting. Out of ~134K numbers, there's only a single duplicate! I'm not sure whether that's a good thing or not, as I haven't profiled any other random number generated in this manner before.

Since I'm planning on taking 1 reading a minute for at least a week (that's 10080 readings), I'm not sure that I'm going to get close to running into this issue - but it's always good to be prepared I guess......

Found this interesting? Got a better way of doing it? Comment below!

Sources and Further Reading

Summer Project Part 1: LoRaWAN Signal Mapping!

What? A new series (hopefully)! My final project for my Masters in Science course at University is taking place this summer, and on the suggestion of Rob Miles I'll be blogging about it along the way.

In this first post, I'd like to talk a little bit about the project I've picked and my initial thoughts.

As you have probably guessed from the title of this post, the project I've picked is on mapping LoRaWAN signal coverage. I'm specifically going to look at that of The Things Network, a public LoRaWAN network. I've actually posted about LoRa before, so I'd recommend you go back and read that post first before continuing with this one.

The plan I've drawn up so far is to build an Internet of Things device with an Arduino and an RFM95 (a LoRa modem chip) to collect a bunch of data, which I'll then push through some sort of AI to fill in the gaps.

The University have been kind enough to fund some of the parts I'll need, so I've managed to obtain some of them already. This mainly includes:

  • Some of the power management circuitry
  • An Arduino Uno
  • A bunch of wires
  • A breadboard
  • A 9V battery holder (though I suspect I'll need a different kind of battery that can be recharged)
  • Some switches

(Above: The parts that I've collected already. I've still got a bunch of parts to go though.)

I've ordered the more specialised parts through my University, and they should be arriving soon:

I'll also need a project box to keep it all in if I can't gain access to the University's 3D printers, but I'll tackle that later.

I'll store on a local microSD card for each message a random id and the location a message was sent. I'll transmit the location and the unique id via LoRaWAN, and the server will store it - along with the received signal strength from the different gateways that received the message.

Once a run is complete, I'll process the data and pair the local readings from the microSD card up with the ones the database has stored, so that we have readings from the 'block spots' where there isn't currently any coverage.

By using a unique random id instead of a timestamp, I can help preserve the privacy oft he person carrying the device. Of course, I can't actually ask anyone to carry the device around until I've received ethical approval from the University to do so. I've already submitted the form, and I'm waiting to hear back on that.

While I'm waiting, I'm starting to set up the backend application server. I've decided to write it in Node.js using SQLite to store the data, so that if I want to do multiple separate runs to compare coverage before and after a gateway is installed, I can do so easily by just moving on to a new SQLite database file.

In the next post, I might talk a little bit about how I'm planning on generating the random ids. I'd like to do some research into the built-in random() function and how ti compares to other unpredictable sources of randomness, such as comparing clocks.

AT24C64 EEPROM and the Arduino

For a project of mine I've bought a bunch of parts. One of those are a bunch of AT24C64 EEPROM chips - which are basically really small SD cards which, in this case, can store 64 KiB of data - even when the power is switched off, as you'd expect.

I ended up having a bit of trouble getting it to work though, as the Arduino IDE appears to have been abandoned and I don't think it's still in development. Still, it works well enough. Anyway, I thought I'd document my findings here for future reference, and to save you a bit of trouble if you find yourself in a similar situation!

The first issue I ran into was in trying to get the associated library to work. I kept getting errors like these:

sketch/eeprom.ino.cpp.o:(.text.loop+0x1c): undefined reference to `AT24CX::writeChars(unsigned int, char*, int)'
sketch/eeprom.ino.cpp.o:(.text.loop+0x20): undefined reference to `AT24CX::readChars(unsigned int, char*, int)'
sketch/eeprom.ino.cpp.o:(.text.loop+0x49): undefined reference to `AT24CX::writeChars(unsigned int, char*, int)'
sketch/eeprom.ino.cpp.o: In function `loop':

Strange. I thought I'd added the #include "lib/AT24Cx/AT24CX.h" to the top? Sure enough, I had. It turns out that the problem actually lay in the fact that I'd used a git submodule to bring in the AT24Cx library, such that the library was not located in the same folder as the .ino file - so the Arduino IDE, in all its wisdom, decided that including the library's .cpp files was hardly necessary O.o

The solution I found was to #include the .cpp explicitly like so:

#include "lib/AT24Cx/AT24CX.cpp"

The other issue I stumbled across was that it couldn't find my EEPROM chip via I2C. Even the demo I2C scanner couldn't find it! It turned out, after searching up a storm on DuckDuckGo, that I needed a pair of 1kΩ resistors stretching from the I2C pins tot he +5V power rail. Here's a diagram I created in Fritzing to show you what I mean:

A circuit diagram showing my wiring for the AT24C64. Note the 1KΩ resistors going form the SCL and SDA pins to the +5V power rail in the breadboard.

(svg, fritzing file)

As usual with the various arduino test programs I find / write, you can get a hold of them in my main arduino repository on my personal git server.

Deep Sleep on ESP-Based Chips

If you're interested in the Arduino ecosystem, you've no doubt come across the Wemos family of boards. Based on the ESP8266, they have WiFi and TCP / UDP support built in! While that's very cool indeed for such a low-power device, in this post I'll be focusing on another cool aspect of the chipset, as I'm going to need to for a project in the nearish future (which I might blog about too!).

Curiously, the ESP chipset carries a unique ability to go into so-called 'deep sleep', which turns off everything but an internal counter, which emits a pulse on a specified pin when the sleep time is up. By wiring this specified pin to the RST (Reset) pin with a jumper cable, we can get it to automagically wake itself up after the deep sleep cycle is completed.

A Wemos with the aforementioned cable running from the D0 pin tot he RST pin.

This is a lot simpler than the sleep modes available on other (non-ESP) chips - which are explained here for those interested. Here's an example:

void setup() {
    Serial.begin(9600);

    Serial.print("Initialising - ");

    pinMode(D0, WAKEUP_PULLUP);

    Serial.println("done");

    Serial.println("Waiting: ");
    for(int i = 0; i < 5; i++) {
        delay(1000);
        Serial.print(".");
    }
    Serial.println();

    Serial.println("Entering deep sleep. Goodnight!");
    ESP.deepSleep(5 * 1000000);
}

void loop() {

}

Nice and easy, right? You can see how you'd need to factor this into the design of your program before you get too far into it. Note that I multiply the number of seconds by 1000000 - this is because the sleep time is specified in microseconds - not milliseconds or seconds.

When in deep sleep, people have managed to reduce it's power consumption down to ~100µA(!) - I'll update this post once I manage to make some measurements of my own.

That's about everything I wanted to mention - just to remind myself on how to do it in a few weeks time :-)

Source and Further Reading

Found this useful? Got a question? Comment below!

An (unscientific) Introduction to I2C

I've recently bought an LCD display for a project. Since I don't have many pins to play with, I ended up buying an I2C-driven display to cut the data pins down to just 2: One for outgoing messages, and one for receiving incoming messages from other devices.

It's taken me some time to get to grips with the idea of I2C, so I thought I'd write up what I've learnt so far here, along with some helpful hints if you run into problems yourself.

In effect, I2C is a wire protocol that allows multiple devices to talk to each other over a single pair of cables. Every I2C device has an 8 bit hardware address burned into it that it uses to address itself - much like the Internet Protocol when it comes to it, actually. Devices can send messages to one another using these addresses - though not all at the same time, obviously!

If you want to talk directly over I2C with a device, then Wire.h is the library you want to use. Normally though, devices will come with their own library that utilises Wire.h and communicates with it for you.

As a good first test to see if I2C is working, I found an I2C scanner that scans for connected devices. Since the address space is so limited, it doesn't take long at all:

/* --------------------------------------
 * i2c_scanner
 * 
 * Version 1
 *  This program (or code that looks like it)
 *  can be found in many places.
 *  For example on the Arduino.cc forum.
 *  The original author is not know.
 * Version 2, Juni 2012, Using Arduino 1.0.1
 *  Adapted to be as simple as possible by Arduino.cc user Krodal
 * Version 3, Feb 26  2013
 *  V3 by louarnold
 * Version 4, March 3, 2013, Using Arduino 1.0.3
 *  by Arduino.cc user Krodal.
 *  Changes by louarnold removed.
 *  Scanning addresses changed from 0...127 to 1...119,
 *  according to the i2c scanner by Nick Gammon
 * 
 * Version 5, March 28, 2013
 *  As version 4, but address scans now to 127.
 *  A sensor seems to use address 120.
 * Version 6, November 27, 2015.
 *  Added waiting for the Leonardo serial communication.
 * This sketch tests the standard 7-bit addresses
 * Devices with higher bit address might not be seen properly.
 */

#include <Wire.h>

void setup()
{
    Wire.begin();

    Serial.begin(9600);
    while (!Serial);             // Leonardo: wait for serial monitor
    Serial.println("\nI2C Scanner");
}

void loop()
{
    byte error, address;
    int nDevices;

    Serial.println("Scanning...");

    nDevices = 0;
    for(address = 1; address < 127; address++ )
    {
        // The i2c_scanner uses the return value of
        // the Write.endTransmission to see if
        // a device did acknowledge to the address.
        Wire.beginTransmission(address);
        error = Wire.endTransmission();

        if (error == 0)
        {
            Serial.print("I2C device found at address 0x");
            if (address<16)
                Serial.print("0");
            Serial.print(address,HEX);
            Serial.println("  !");

            nDevices++;
        }
        else if (error==4)
        {
            Serial.print("Unknown error at address 0x");
            if (address<16)
                Serial.print("0");
            Serial.println(address,HEX);
        }
    }
    if (nDevices == 0)
    Serial.println("No I2C devices found\n");
    else
    Serial.println("done\n");

    delay(5000);           // wait 5 seconds for next scan
}

As the initial comment mentions, I can't claim ownership of this code! I got it from here.

With the code in mind, it's time to look at the circuit design.

A simple I2C circuit connecting an Arduino Uno v3 and an LCD display.

(Above: A simple I2C circuit. Credits go to openclipart.org for the images.)

The above connects an Arduino Uno version 3 with a simple LCD display via a breadboard to allow for expansion to connect future devices. The power (the red and blue cables) link the 5V and GND pins from the Arduino to the appropriate pins on the back of the LCD (the image of an LCD I found didn't have the pins showing :P), and the I2C pins (green and yellow) connect the SDA and SCL pins on the Arduino to the LCD display.

With the circuit done, that completes the system! All that remains now is to build something cool with the components we've put together :D

The final product of the above IRL!

PixelBot Part 2: Devices need protocols, apparently

A selection of the technologies I'm using to put together my PixelHub server.

So there I was. I'd just got home, turned on my laptop, opened the Arduino IDE and Monodevelop, and then.... nothing. I knew I wanted my PixelBot to talk to the PixelHub I'd started writing, but I was confused as to how I could make it happen.

In this kind of situation, I realised that although I knew what I wanted them to do, I hadn't figured out the how. As it happens, when you're trying to get one (or more, in this case) different devices to talk to each other, there's something rather useful that helps them all to speak the same language: a protocol.

A protocol is a specification that defines the language that different devices use to talk to each other and exchange messages. Defining one before you start writing a networked program is probably a good idea - I find particularly helpful to write a specification for the protocol that the program(s) I'm writing, especially if their function(s) is/are complicated.

To this end, I've ended up spending a considerable amount of time drawing up the PixelHub Protocol - a specification document that defines how my PixelHub server is going to talk to a swarm of PixelBots. It might seem strange at first, but I decided on a (mostly) binary protocol.

Upon closer inspection though, (I hope) it makes a lot of sense. Since the Arduino is programmed using C++ as it has a limited amount of memory, it doesn't have any of the standard string manipulation function that you're used to in C♯. Since C++ is undoubtedly the harder of the 2 to write, I decided to make it easier to write the C++ rather than the C&sharp. Messages on the Arduino side are come in as a byte[] array, so (in theory) it should be easy to pick out certain known parts of the array and cast them into various different fundamental types.

With the specification written, the next step in my PixelBot journey is to actually implement it, which I'll be posting about in the next entry in this series!

My PixelBot is Connected! (Part 1)

After many attempts and solving many problems, I've finally gotten my Wemos-powered PixelBot to connect via TCP to a C# server component I've written. Since I experienced so many problems with it, I decided to post about it here to help others who want to do the same as I.

The first (and arguably most difficult) hurdle I came across was the lack of correct information. For one, the TCP client class is actually called WifiClient (which is confusing in and of itself). For another, there aren't any really good tutorials out there that show you what to do.

In addition, I wanted to build a (rather complicated as it turns out!) auto discovery system to allow my PixelBot to find the PixelHub (the server) automatically. As usual, there was even less information about this task! All I found was an outdated guide and rather simplistic example. I ended up inspecting the header file of the WiFiUDP class in order to figure it out.

In this post, I'm going to explain how I got the autodiscovery mechanism working. Before we begin, you need to understand what multicast UDP is and how it works. For those of you who don't, I've posted all about it.

There are several ways that one can go about writing an autodiscovery mechanism. While Rob Miles chose mDNS, I ended up approaching the problem from a different angle and writing my own protocol. It's best explained with a diagram:

A diagram explaining the PixelBot's autodiscovery mechanism.

  1. The PixelHub server has a beacon that sends out pings to tell the PixelBots where it is
  2. The PixelBots subscribe to the beacon ping channel when they want to find the server
  3. The PixelBots decode the incoming ping packets to find the discover of the server
  4. The PixelBots connect to the PixelHub server using the credentials that it found in the beacon pings it decoded!

In order to receive these beacon pings, we need to set up a UDP listener and wire it up to receive multicast packets. In the following example I'm multicasting on 239.62.148.30 on port 5050.

IPAddress beaconAddress = IPAddress(239, 62, 148, 30);
unsigned int beaconPort = 5050;

WiFiUDP UdpClient;
UdpClient.beginMulticast(WiFi.localIP(), beaconAddress, beaconPort);

After connecting to the multicast address, we then need to receive the pings:

// Create a buffer to hold the message 
byte datagramBuffer[datagramBufferSize];
// Prefill the datagram buffer with zeros for protection later
memset(datagramBuffer, '\0', datagramBufferSize);
while(true) {
    int datagramSize = UdpClient.parsePacket();
    Serial.print("Received datagram #");
    Serial.print(datagramSize);
    Serial.print(" bytes in size from ");
    Serial.print(UdpClient.remoteIP());
    Serial.print(":");
    Serial.print(UdpClient.remotePort());

    // Don't overflow the message buffer!
    if(datagramSize > datagramBufferSize) {
        Serial.println(", but the message is larger than the datagram buffer size.");
        continue;
    }
    // Read the message in now that we've verified that it won't blow our buffer up
    UdpClient.read(datagramBuffer, datagramSize);

    // Cheat and cast the datagram to a character array
    char* datagramStr = (char*)datagramBuffer;

}

That's rather complicated for such a simple task! Anyway, now we've got our beacon ping into a character array, we can start to pick it apart. Before we do, here's what my beacon pings look like:

server@10.0.40.66:5050
  ^        ^        ^
 Role IP Address   Port

Although it looks simple, in C++ (the language of the arduino) it's rather a pain. Here's what I came up with:

// Define the role of the remote server that we're looking for
char desiredRemoteRole[7] = { 's', 'e', 'r', 'v', 'e', 'r', '\0' };

// Find the positions of the key characters
int atPos = findChar(datagramStr, '@');
int colonPos = findChar(datagramStr, ':');

// Create some variables to store things in
char role[7];
char serverIp[16];
char serverPortText[7];
int serverPort = -1;
// Fill everything with zeroes to protect ourselves
memset(role, '\0', 7);
memset(serverIp, '\0', 16);
memset(serverPortText, '\0', 7);
// Extract the parts of the 
strncpy(role, datagramStr, atPos);
strncpy(serverIp, datagramStr + atPos + 1, colonPos - atPos - 1);
strncpy(serverPortText, datagramStr + colonPos + 1, datagramSize - colonPos - 1);

Serial.println("complete.");

// Print everything out to the serial console
Serial.print("atPos: "); Serial.println(atPos);
Serial.print("colonPos: "); Serial.println(colonPos);

Serial.print("Role: "); Serial.print(role); Serial.print(" ");
Serial.print("Remote IP: "); Serial.print(serverIp); Serial.print(" ");
Serial.print("Port number: "); Serial.print(serverPortText);
Serial.println();

// If the advertiser isn't playing the role of a server, then we're not interested
if(strcmp(role, desiredRemoteRole) != 0)
{
    Serial.print("Incompatible role "); Serial.print(role); Serial.println(".");
    continue;
}
Serial.println("Role ok!");
serverPort = atoi(serverPortText);

Phew! That's a lot of code! I've tried to annotate it the best I can. The important variables in the are serverIp the serverPort - they hold the IP address and the port number of the remote machine that we want to connect to.

I hope that's somewhat helpful. If there's anything you don't understand or need help with, please post a comment down below :-)

Next time, I'm going to show off the TCP connection bit of the system. Stay tuned :D

Art by Mythdael