Starbeamrainbowlabs

About

Hello!

I am a computer science student researcher who is doing a PhD at the University of Hull. My project title is Using Big Data and AI to dynamically predict flood risk.

I started out teaching myself about various web technologies, and then I managed to get a place at University, where I am now. I've previously done a degree (BSc Computer Science) and a Masters (MSc Computer Science with Security and Distributed Computing) at the University of Hull. I've done a year in industry too, which I found to be particuarly helpful in learning about the workplace and the world.

I currently know C# + Monogame / XNA (+ WPF), HTML5, CSS3, Javascript (ES7 + Node.js), PHP, C / C++ (mainly for Arduino), some Rust, and Python. Oh yeah, and I can use XSLT too.

I love to experiment and learn about new things on a regular basis. You can find some of the things that I've done in the labs and code sections of this website, or on GitHub (both in my personal time and for my PhD). My current big personal projects are Pepperminty Wiki, an entire wiki engine in a single file (the source code is spread across multiple files - don't worry!), and WorldEditAdditions. Nibriboard (a multi-user real-time infinite whiteboard), although the latter is in its very early stages. -->

I can also be found in a number of other different places around the web. I've compiled a list of the places that I can remember below.

I can be contacted at the email address webmaster at starbeamrainbowlabs dot com. Suggestions, bug reports and constructive criticism are always welcome.

For those looking for my GPG key, you can find it here. My key id is C2F7843F9ADF9FEE264ACB9CC1C6C0BB001E1725, and is uploaded to the public keyserver network, so you can download it with GPG like so: gpg --keyserver hkps://keyserver.ubuntu.com:443/ --recv-keys C2F7843F9ADF9FEE264ACB9CC1C6C0BB001E1725

Blog

Blog Roll | Article Atom Feed | An image of a white letter, representing an email.Mailing List


Latest Post

Autoplant, Part 1: Overview

At a recent plant sale at my University, I bought myself both a parlour palm and an areca palm for my desk. They look lovely, but I always worry that I'm going to forget to water them.

The palms on my desk, with a blurred background

Having some experience with arduino already, I decided to resolve the issue by implementing an arduino-based system to monitor my new plants and log the resulting data in my existing collectd-based (see also CGP which I use, but sadly it's abandonware. Any suggestions for alternatives are greatly appreciated) monitoring system I use for the various servers and systems that I manage.

The eventual aim is to completely automate the watering process too - but before I can do that I need to first get the monitoring up and running so that I can calibrate the sensors, so this is what I'll be focusing on in this post.

Circuit design

To do this, I've used a bunch of parts I have lying around (plug a few new ones), and wired up a NodeMCU v0.9 to some sensors:

The circuit I've designed. See below for explanation.

(Above: The circuit I've wired up. See the Fritzing file.)

A full list of parts can be found at the end of this post, along with links to where I got them from. The sensors I'm using are:

  • 2 x Capacitive soil moisture sensors
  • 2 x Liquid level sensors
  • 1 x BME280 that I had lying around and thought "why not?"

Both the soil sensors and the liquid level sensors give out an analogue signal (shown in orange), but unfortunately the NodeMCU v0.9 (based on the ESP8266) only has a single analogue pin, so I bought myself a CD4051 from switch electronics (link at the bottom of this post) to act as a multiplexer. Given 3 digital wires to act as a channel selector (shown in purple), it allows you to select between 8 different analogue channels - perfect for my use-case. You can see it in the diagram at the left-hand side of the larger breadboard.

While the Fritzing design for a USB breakout board isn't exactly the same as the ones I have, I plan on using them to transport power, ground, and the 2 analogue outputs from the 2 sensors for the plant on the other side of my desk.

The other component here is a BME280 I have lying around to monitor temperature, humidity, and air pressure. This isn't required, but since I have one lying around anyway I thought it was a good idea to use it.

Networking

With the circuit designed and implemented (still got to finalise the USB breakout boards), the next thing to organise is the transport and logging for the data generated. MQTT is easy to use on the Arduino because of PubSubClient, so that's what I decided on using.

Time for another diagram!

A brief overview of the networking setup I'll be going for with the backend. Subject to change.

(Can't see the above? Try a PNG instead.)

If you've followed my blog here for a while, you might remember that I have a cluster that's powered by a bunch of Raspberry Pis running Hashicorp Nomad and Consul.

On this cluster - amongst a many other things - I have an MQTT server (check out my earlier post on how to set one up) running inside a Docker container as a Nomad task. Connections to my Mosquitto MQTT server are managed by Fabio, which reverse-proxies connections to Mosquitto. Fabio exposes2 ports by which one can connect to the MQTT server:

  • TCP port 1883: Unencrypted MQTT
  • TCP port 8883: (TLS encrypted) MQTTS

In doing so, Fabio terminates TLS so that Mosquitto doesn't need to have access to my TLS certificates. For those interested, the proxy.addr in your fabio.properties file is actually a comma-separated list, and to achieve TLS termination for MQTTS you can add something like this to proxy.addr (make sure to use an up-to-date cipher list):

:1883;proto=tcp,:8883;proto=tcp;cs=mooncarrot;tlsmin=tls12;tlsciphers="TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

Then if we continue working backwards, the next piece of the puzzle is port forwarding on my router. While Fabio exposes both MQTT and MQTTS, I only port-forward MQTTS and not unencrypted MQTT. My autoplant system will only be communicating over MQTTS, so it doesn't make sense to expose the less secure unencrypted port to the world too.

From here, no matter where in the world my autoplant monitoring system is located it will still be able to send sensor values back to be logged.

Working forwards again from the Mosquitto MQTT server, the Arduino is going to send MQTT messages to the sensors/data topic with messages that look something like this (but minified):

{
    "id": "plant-a",
    "sensor": "soil",
    "value": 2.41
}

The MQTT plugin for Collectd doesn't support JSON input though, so I'm going to write a broker that translates messages from the JSON format above to the format that Collectd likes. My reasoning for this is twofold:

  1. I also want to log data to a tab-separated-values file for later long-term analysis
  2. The collectd format is somewhat complicated and has the potential to mess up other parts of my monitoring system, so I want to do some extra validation on incoming data from remote sensors

I haven't fully decided on which language I'm going to write this validating broker in, but I'm thinking it might end up being a shell script. Weird as it sounds, I found a cool MQTT library for Bash called bish-bosh which I want to try out. My second choice here would be Rust, but I unfortunately can't find a (preferably pure-rust) MQTT(S) library, which I'm finding rather strange.

Either way, if possible I'm going to package up the completed implementation of this broker into a Docker container and write a quick Hashicorp Nomad job file to get it running on my cluster so that it benefits from the reundancy of my Nomad cluster. Then, with collectd listening on another topic, it can transparently bridge the 2. I'm not quite sure how will collectd's MQTT plugin actually works though, so this shell script may end up being a child process of my main collectd server using the exec plugin instead.

Conclusion

In this post, I've outlined my solution to a seemingly simple problem of watering plants automatically (because all simple problems need complex solutions :P). With an arduino-based system, I'm going to send messages via MQTTS containing sensor data, which will then be passed to a backend for processing. In future posts in this series, I want to (in no particular order):

  • Go through the code for the arduino I've written
  • Look at the code I have yet to write for the translation broker
  • Explore the 3d printed parts I have yet to design for various purposes
  • Show off the final completed circuit in action
  • Look at some initial statistics collected by the sensors
  • Start to play with pumping water around and different holding containers / designs, etc

Not all of these warrant a separate post, but there are definitely multiple posts to come on this project :D

Full parts list

  • 1 x NodeMCU v0.9 (anything with WiFi will do, I just had an old on lying around)
  • 1 x Half size breadboard
  • 1 x Mini breadboard (the smallest I had)
  • 2 x Capacitive soil moisture sensor (source, significantly cheaper if you're willing to wait ages)
  • 2 x Waveshare liquid level sensor (source, original website)
  • 1 x CD4051 8:1 analogue multiplexer (source - breadboard compatible)
  • 2 x USB type a breakouts
  • 1 x BME 280 (I got mine cheap on AliExpress)
  • Lots of jumper wires

By on

Labs

Code

Tools

I find useful tools on the internet occasionally. I will list them here.

Art by Mythdael