Starbeamrainbowlabs

Stardust
Blog


Archive

Mailing List Articles Atom Feed Comments Atom Feed Twitter Reddit Facebook

Tag Cloud

3d account algorithms announcement architecture archives arduino artificial intelligence artix assembly async audio bash batch blog bookmarklet booting c sharp c++ challenge chrome os code codepen coding conundrums coding conundrums evolved command line compilers compiling css dailyprogrammer debugging demystification distributed computing downtime electronics email embedded systems encryption es6 features event experiment external first impressions future game github github gist graphics hardware hardware meetup holiday html html5 html5 canvas infrastructure interfaces internet io.js jabber javascript js bin labs learning library linux low level lua maintenance manjaro network networking node.js operating systems performance photos php pixelbot portable privacy programming problems project projects prolog protocol protocols pseudo 3d python reddit reference release releases resource review rust secrets security series list server software sorting source code control statistics svg technical terminal textures three thing game three.js tool tutorial tutorials twitter ubuntu university update updates upgrade version control virtual reality virtualisation visual web website windows windows 10 xmpp xslt

Pepperminty Wiki CLI

The Pepperminty Wiki CLI. in a terminal window, with a peppermint overlaid in the top left of the image.

I've got a plan. Since I'm taking the Mobile Development module next semester, I'd like to create an Android app for Pepperminty Wiki that will let me edit one or more instances of Pepperminty Wiki while I'm, say, on a bus.

To this end, I'll need to make sure that Pepperminty Wiki itself is all ready to go - which primarily entail making sure that its REST API is suitably machine-friendly, so that I can pull down all the information I need in the app I build.

Testing this, however, is a bit of a challenge - since I haven't actually started the module yet. My solution, as you might have guessed by the title of this blog post, is to build a command-line interface (CLI) instead. I've been writing a few bash scripts recently, to I tried my hand at creating something that's slightly more polished. Here's a list of the features supported at the time of posting:

  • Listing all pages
  • Viewing a specific page
  • Listing all revisions of a page
  • Viewing a specific revision of a page

Support for searching is on the cards, but it's currently waiting on support for grabbing search results as json / plain text from Pepperminty Wiki itself.

I'll be updating it with other things too as I think of them, but if you'd like to give it a try now, then here's the source:

It should update dynamically as I update the script. Simply save it to a file called peppermint - and then you can run ./peppermint to get an overview of the commands it supports. To get detailed help on a specific command, simply run ./peppermint {command_name} to get additional help about that specific command - and additional help for that command will be displayed if it supports any further arguments (it will be executed directly if not).

Sound interesting? Any particular aspect of this script you'd like explaining in more detail? Want to help out? Leave a comment below!

The Great Migration of Manjaro

The artix linux logo. It's relevance will become clear by the end! :P

It was just before lunch in the library, and I was checking my university emails on my travelling laptop that runs Manjaro OpenRC. While that was going on, I was inducing a few updates that it notified me about with yaourt -Syua. First mistake.

During the installation, it decided to upgrade openrc to the Briston in the AUR (Arch User Repository), but I didn't think anything of it particularly - I knew that Manjaro OpenRC was dying deprecated. Second mistake.

Once the updates were complete, I shut it down and sent on my way - or at least I tried to - it wouldn't shut down, instead proceeding to log out and leave it at that. I resolved to investigate the problem when I got home. Third mistake.

By the time I came to use it again, I was greeted with an ominous message:

[Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0x52 (or later)
Failed to execute /init (error -2)
Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.13.2-1-MANJARO #1
Hardware name: Entroware Apollo/Apollo, BIOS 1.05.05 04/27/2017

Hrm. That's odd. Maybe something went wrong in the update? Linux has what's called kernel parameters that tell it how to boot. They specify things like "here's the root partition of the system", and "please let me edit files on the system after booting". To udnerstand how this fits into the next part of the story, it's first necessary to take a quick retour and look at how, precisely the linux kernel goes about booting a system. This is best explained with a diagram:

The linux kernel boot process.

(Rendered with Ascidia. Textual diagram source available here)

  1. BIOS / UEFI POST - The starting point of the boot process. The BIOS / UEFI turns on all the devices, runs some basic hardware checks, and (usually) gives the user a choice of what they want to boot from.
  2. rEFInd - grub may be used instead of rEFInd, but the basic principle is the same: it asks the user how they want to boot from the hard drive. Kernel parameters are decided on here.
  3. Initialisation: The Linux kernel is executed by the bootloader, and it proceeds to initialise itself and the connected devices.
  4. Mount initial RAM disk: The Linux encounters a chicken-and-egg problem rather early on: How can it start talking to the connected devices if it doesn't know how to talk to them? The initial RAM disk solves that problem: It contains a bunch of drives and other such components that the kernel needs to initialise all the connected devices. It's like a cut-down root file system, in a sense.
  5. Load drivers: The Linux kernel loads the drivers from the initial RAM disk (aka initrd) and starts initialising all the connected devices.
  6. Mount root (read-only): The main root file system is mounted next, but only in read-only mode while the boot process finishes.
  7. Execute init: It is at this point that the very first process is executed. It usually presides at /sbin/init, but this can be changed through the init kernel parameter.
  8. Mount root (read-write): The init process (under SysVinit at least) then remounts the root filesystem such that it is writeable.
  9. Mount other partitions: The next job is the mounting of the other partitions in /etc/fstab. This is also done by SysVinit if I recall correctly.
  10. Reach runlevels: The main runlevels managed by the service manager (e.g. OpenRC) are now executed in order by the service manager.

Phew, that took more explaining than I thought! And to think it all happens in the span of about 10 seconds....! With that out of the way, let's continue with the story.

Let's try specifying the init kernel parameter - maybe the update cleared it for some random reason....? I had no idea what I was getting myself into :P

Unexpectedly, specifying init=/sbin/init didn't work. Neither did specifying init=/bin/sh. At this point, I suspected that there was something seriously wrong. I (correctly) guessed that it was the update I performed that morning that was to blame. After a bunch of backing and forthing, I managed to get hold of a previous copy of the openrc package that was replaced by the 0.27 version from the AUR. After doing a full backup, I tried installing it and removing the new openrc-sysvinit package that was also installed.

Before we continue further, I should probably explain how I managed to install the previous package version. Didn't I just explain that my system wasn't bootable? Well, yes. But I also had the original manjaro-architect installation media that I used to build the system in the first place. With that in hand, I could use rEFInd to boot from that (my UEFI firmware makes it a bit of a pain otherwise!), and then mount the root partition of the broken system and chroot into it. This process allows me to pretend that the system is actually booted, while piggybacking off the live installation media of the boot process. It works a bit like this:

lsblk # Find the root partition
mkdir /mnt/os;
mount /dev/sdZY /mnt/os # Mount the root partition
mount /dev/sdAB /mnt/os/boot/efi # Mount the EFI partition
manjaro-chroot /mnt/os bash # Enter the chroot and execute bash

Back to the story. Sadly, valiant though my effort was to replace the openrc and openrc-sysvinit packages was, it did not solve the problem. Eventually, I ended up having to perform a blind migration to Artix Linux, the spiritual successor to both Manjaro OpenRC and Arch OpenRC (apparently the developers of both came together to create Artix Linux).

Eventually, I ended up with a successful migration that I performed inside the chroot, and the system was bootable again! Next time, I'll always run pacman -Syu before yaourt -Syua. I'll also set up a temporary backup solution for my system files (I've already got one in place for my personal files) while I figure out a more permanent one that backs up across the network.

Sources and Further Reading

Run a program on your dedicated AMD graphics card on Linux

I've recently figured out how to run a program on my dedicated AMD R7 M445 graphics card in Ubuntu 17.04, and since it's taken me far too long to around figuring it out, I thought I'd note it down here for future reference - if it helps you too, let me know in the comments below!

It's actually really simple. First, check that your dedicated AMD graphics card shows up with lspci:

lspci

If it's anything like my setup, you'll get a pair of rows like this (though they might not be next to each other):

00:02.0 VGA compatible controller: Intel Corporation HD Graphics 620 (rev 02)
01:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Topaz XT [Radeon R7 M260/M265 / M340/M360 / M440/M445] (rev c3)

Thankfully, my dedicated AMD card is showing (better than it did in previous versions of ubuntu, too, which thought it was an M225!). Next, we need to check that the amdgpu kernel module is loaded with a quick lsmod:

lsmod | grep -i amd

On my laptop, I get this:

amdkfd                139264  1
amd_iommu_v2           20480  1 amdkfd
amdgpu               1564672  1
i2c_algo_bit           16384  2 amdgpu,i915
ttm                    98304  1 amdgpu
drm_kms_helper        151552  2 amdgpu,i915
drm                   352256  9 amdgpu,i915,ttm,drm_kms_helper

Yay! It's loaded. Now to do a test to see if we can run anything on it:

glxinfo | grep "OpenGL renderer"
DRI_PRIME=1 glxinfo | grep "OpenGL renderer"

The above runs glxinfo twice: Once on the integrated graphics card, and once on the dedicated graphics card. The key here is the DRI_PRIME=1 environment variable - this tells the amdgpu driver that this process should run on the dedicated graphics and not the integrated graphics card. On my machine, I get this output:

OpenGL renderer string: Mesa DRI Intel(R) HD Graphics 620 (Kabylake GT2) 
OpenGL renderer string: Gallium 0.4 on AMD ICELAND (DRM 3.9.0 / 4.10.0-33-generic, LLVM 4.0.0)

As you can see, the latter invocation of the command ran on the dedicated AMD graphics card, and the former on the integrated graphics. So simple!

Now that we've verified that it works, we can do it with any program:

DRI_PRIME=1 inkscape

Did this you find this helpful? Did it work (or not)? Let me know in the comments!

Sources

Unmounting NFS Shares on Shutdown in OpenRC Manjaro

A cool SVG of a server. (Above: A clipart image of a server. Source)

Since I've been using Manjaro with OpenRC when I'm out and about, I've been steadily fixing little issues and niggles I've been encountering one by one (such as finding the option to let you move the windows on the taskbar panel around yourself).

One of the first issues I encountered was that OpenRC would generously take the network down before my NFS (network file system) shares have been unmounted. This results in lengthly delays when shutting down as each of the components of the NFS mounting system have to be waited upon by OpenRC and finally killed after taking too long to shut down.

Initially I attempted to investigate reordering the shutdown process, but that quickly grew out of hand as I was investigating, and I discovered that it was not a particularly practical or, indeed, stable solution to my particular problem. Next, I found autofs which looked like it would solve the problem by automatically mounting and unmounting my NFS shares as and when they are needed, but despite assisance from someone far more experienced in the Manjaro world than I (thank you!) couldn't get it to work reliably. In addition, it started exhibiting some odd behaviour like hiding all my other mounts in my /media folder, so I went on the hunt for better solution.

Quite by chance (all thanks to Duck Duck Go Instant Answers!) I stumbled upon NetworkManager dispatcher scripts. NetworkManager is the service / application that manages, surprisingly, the network connections on several major linux distributions - including Ubuntu (which I've used before), and, crucially, Manjaro. Although the answer said that the functionality I wanted had been removed, upon looking into the amtter it appeared to be an artifact of the way systemd shutdown the system, and so I gave it a whirl anyway just to see if it would work.

Thankfully it did end up working! To that end, I thought I'd (re)post the solution I found here for future reference, and in case it helps anyone else :-)

Assuming you already have your shares set up and working in your /etc/fstab, you can create a file in the folder /etc/NetworkManager/dispatcher.d/pre-down.d with the contents something like this:

#!/bin/sh

logger "Unmounting NFS shares gracefully before the network goes down...";

umount /media/bob/rocket-diagrams-nas;
umount /media/sean/satellite-schematics;

logger "Unmounted NFS shares successfully.";

Once done, you'll need to make it executable with a quick sudo chmod +x, and try rebooting to test it!

In theory, this could be used to do other things that need to be done before the network is taken down, like making a sekret tracking request to your web server for anti-theft purposes, or uploading a backup of your laptop's /etc directory automagically in case it comes to a sticky end.

Sources and Further Reading

Debug your systemd services with journalctl

Some nice bulbs and bits of wood.

The chances are that if you're using linux, you will probably have run into systemd. If you find yourself in the situation where you've got a systemd service that keeps dying and you don't know why (I've been there before several times!), and there's nothing helpful in /var/log, before you give up, you might want to give journalctl a try. It's systemd's way of capturing the output of a service and storing it in it's logging system (or something).

When I first found out about it, I read that apparently journalctl -xe servicename would show me the logs for any given service. It turned out that it wasn't the case (it just threw a nasty error), so I went trawling through the man pages and found the correct command-line switch. If you've got a service called rocketbooster.service, and you want to see if systemd has any logs stored for it, then you can execute this command:

journalctl --unit rocketbooster.service

...or for short

journalctl -u rocketbooster.service

It should open the logs (if there are any) in less - with the oldest logs at the top, so you might need to scroll all the way down to the bottom to see anything that's relevant to your problem (shift + G will take you to the bottom of the file).

I've found that systemd has a habit of rotating the logs too - and journalctl doesn't appear to know how to access the rotated logs, so it's best if you use this command as soon as possible after failure (suggestions on how to access these rotated logs are welcome! Post down in the comment :D).

I thought I'd document it here in case it was useful to anyone - and so I don't forget myself! :P

Access your home linux box from anywhere with SSH tunnels

An abstract tunnel that doesn't hold much relevant to the blog post :P

(Header by GDJ from openclipart.org. Source page)

....and other things! Recently, I bought a Raspberry Pi 3. Now that the rest of the components have arrived, I've got a rather nice little home server that's got a 1 terabyte WD PiDrive attached to it to provide lots of lovely shared storage, which is rather nice.

However, within a few weeks I was faced with a problem. How do I access my new box to configure it from my internship when I'm on lunch? Faced with such a challenge, I did what anyone would, and took to the internet to find a solution.

It didn't take long. A while ago I heard about these things called 'SSH tunnels', which, while not designed for a high throughput, are more than adequate for a low-intensity SSH connection that runs a few kilobytes a second in either direction. After reading this excellent answer by erik on the Unix & Linux StackExchange, I had an understanding of how SSH tunnels work, and was ready to put together a solution. You should go and read that answer if you'd like to understand SSH tunnels too - it explains it much better than I ever could :P

With that knowledge in hand, I went about planning the SSH tunnel. I already have a server a public IP address (it's hosting this website!), so I needed a reverse tunnel to allow me to access a port local to my linux box at home (called elessar - a virtual cookie for anyone who gets the reference!) from starbeamrainbowlabs.com.

Important! Ask yourself whether it's moral and ethical to set up an ssh tunnel before you think about following along with this article! If you find yourself behind a firewall or something similar, then the chances are that it's there for a good reason - and you might get into trouble if you try and circumvent it. I won't be held responsible for any loss or damages of any description caused by the reading of this post.

First job: create a limited account on starbeamrainbowlabs.com for elessar to SSH into. That's easy:

sudo useradd --system ssh-tunnel

Then, with a few quick lines in /etc/ssh/sshd_config:

Match User ssh-tunnel
    ForceCommand echo 'This account can only be used for ssh tunnelling.'

....we can prevent the ssh-tunnel user from being abused to gain shell access to the server (let me know if there are any further measures I can put in place here).

Now that I had a user account to ssh in as, I could set up a public / private keypair to authenticate with starbeamrainbowlabs.com, and cook up an SSH command for elessar that would set up the appropriate tunnel. After fiddling around a bit, I came up with this that did the job:

ssh -TN -R30582:localhost:5724 ssh-tunnel@starbeamrainbowlabs.com

Very cool. So with that command executing on elessar, I could ssh into elessar from starbeamrainbowlabs.com! In short, it sets up a tunnel that will make port 30582 on starbeamrainbowlabs.com tunnel through to port 5724 on elessar - the port on elessar that has SSH running on it, without allocating a pseudo-tty to save resources. explainshell.com can, well, explain it in more detail if you're interested.

Having an SSH command that would set up the tunnel is nice, but it's not very useful, since I have to execute it first before I can actually SSH into elessar from afar.

The solution was actually a little bit complicated. First, I wrote a simple systemd service file (systemd is what I have installed, since it's vanilla raspbian - this should be easily adaptable to other systems and setups) to start the SSH tunnel automagically on boot:

[Unit]
Description=SSH tunnel from starbeamrainbowlabs.com to local ssh server.

[Service]
Type=simple
ExecStart=/usr/bin/ssh -TN -R30582:localhost:5724 ssh-tunnel@starbeamrainbowlabs.com

[Install]
WantedBy=network-online.target

I quickly realised that there were a few flaws with this approach. Firstly, it tried to start the SSH connection before my router had connected to the internet, since my router starts faster than the box that initialises the fibre connection to my ISP. Secondly, it fails to retry when the connection dies.

The first problem can be solved relatively easily, by wrapping the ssh command in a clever bit of shell scripting:

/bin/sh -c 'until ping -c1 starbeamrainbowlabs.com &>/dev/null && sleep 5; do :; done && /usr/bin/ssh -TN -R30582:localhost:5724 ssh-tunnel@starbeamrainbowlabs.com

The above tries to ping starbeamrainbowlabs.com every 5 seconds until it succeeds, and only then does it attempt to open the SSH connection. This solves the first problem. To solve the second, we need to look at autossh. Autossh is a small tool that monitors an ssh connection in a variety of configurable ways and restarts the connection if ever dies for whatever reason. You can install it with your favourite package manager:

sudo apt install autossh

Substitute apt with whatever package manager you use on your system. With it installed, we can use a command like this:

autossh -o "UserKnownHostsFile /home/ssh-tunnel/.ssh/known_hosts" -o "IdentityFile /home/ssh-tunnel/.ssh/ssh-tunnel_ed25519" -o "PubkeyAuthentication=yes" -o "PasswordAuthentication=no" -o "ServerAliveInterval 900" -TN -R30582:localhost:5724 -p 7261 ssh-tunnel@starbeamrainbowlabs.com

to automatically start our ssh tunnel, and restart it if anything goes wrong. Note all the extra settings I had to specify here. This is because even though I had many of them specified in ~/.ssh/config for the ssh-tunnel user, because of systemd's weird environment when it starts a service, I found I had to specify everything in the command line with absolute paths (ugh).

Basically, the above tells autossh where the known_hosts file is (important for automation!), that it should only attempt public / private keypair authentication and not password authentication, that it should check the server's still there every 15 minutes, and all the other things we figured out above.

Finally, I combined the solutions I came up with for both problems, which left me with this:

[Unit]
Description=SSH tunnel from starbeamrainbowlabs.com to local ssh server.

[Service]
Type=simple
ExecStart=/bin/sh -c 'until ping -c1 starbeamrainbowlabs.com &>/dev/null && sleep 5; do :; done && /usr/bin/autossh -o "UserKnownHostsFile /home/pi/.ssh/known_hosts" -o "IdentityFile /home/pi/.ssh/ssh-tunnel_ed25519" -o "PubkeyAuthentication=yes" -o "PasswordAuthentication=no" -o "ServerAliveInterval 900" -TN -R30582:localhost:5724 -p 7261 ssh-tunnel@starbeamrainbowlabs.com'

[Install]
WantedBy=network-online.target

Here's a version that utilises the -f parameter of autossh to put the autossh into the background, which eliminates the sh parent process:

[Unit]
Description=SSH tunnel from starbeamrainbowlabs.com to local ssh server.

[Service]
Type=forking
Environment=AUTOSSH_PIDFILE=/var/run/sbrl-ssh-tunnel/ssh-tunnel.pid
PIDFile=/var/run/sbrl-ssh-tunnel/ssh-tunnel.pid
ExecStartPre=/bin/mkdir -p /var/run/sbrl-ssh-tunnel
ExecStartPre=-/bin/chown ssh-tunnel:ssh-tunnel /var/run/sbrl-ssh-tunnel
ExecStart=/bin/sh -c 'until ping -c1 starbeamrainbowlabs.com &>/dev/null && sleep 5; do :; done && /usr/bin/autossh -f -o "UserKnownHostsFile /home/pi/.ssh/known_hosts" -o "IdentityFile /home/pi/.ssh/ssh-tunnel_ed25519" -o "PubkeyAuthentication=yes" -o "PasswordAuthentication=no" -o "ServerAliveInterval 900" -TN -R30582:localhost:5724 -p 7261 ssh-tunnel@starbeamrainbowlabs.com'

[Install]
WantedBy=network-online.target

I ended up further modifying the above to set up an additional tunnel to allow elessar to send emails via the postfix email server that's running on starbeamrainbowlabs.com. Let me know if you'd be interested in a tutorial on this!

Sources and Futher Reading

Learn your terminal (or command line)

Enter stage left: the terminal (or command line, on windows). That window with strange white text on a black background. You might not see it, but every operating system has one - humming away in the background, just waiting to be used, but epic arcane skills are needed to navigate this bizarre and perhaps dated window into your computer.... or so it seems.

When you think of your computer, you will probably think of a GUI (a.k.a. goo-ey), with windows, a cursor, and perhaps a few buttons. GUIs make it easy for newcomers to easily find their way around a computer by referencing things that exist in the real world (e.g. folders and files, a floppy disk on the save button, etc.), but they can be inherently slower to use - especially for long series of perhaps repetitive tasks that stay essentially the same.

A terminal (linux and friends) or a command line (windows) is another view into your computer. It's a way of controlling your computer with text. Text that follows particular set of rules, that can be saved and repeated at will through the use of scripts. It's built on commands, each of which does one thing and one thing well. On their own they're mildly useful, but together they form a powerful framework that can perform almost any task. It's certainly different (and there's a little bit of learning curve, to be sure), but not as hard or arcane as you might think it currently.

A knowledge of the terminal or command line on your computer can be rather useful - especially so for those involved in computer science or technical support. How long would it take you to flatten a large set of deeply nested folders with a GUI? Or convert and recompress few folders worth of videos? Or even renew all your ssl certificates on your web server? All of these things can be automated through the use of a terminal or command line.

Even if you're just a casual computer user who's not into programming, it's still worth at least looking into. Perhaps it'll save you some time! Perhaps it'll save you from asking your friend where something is on their computer when you can't find it. Maybe it'll even save you if your computer suddenly decides it doesn't want to boot up properly. And you'll look cool doing it too :P (What better reason is there?)

If I've somehow managed to convince you to dive in and take up the challenge learning, then I'll end this somewhat different post with a collection of places you can go to get started.

Make your linux learning experience painless with tldr-pages!

If you've been learning linux for a little while, you'll probably have encountered man pages. They are the complete documentation of all the tools, commands(, and kernel functions) available on the system you're currently on (read them online here!). If you have encountered them, you'll also know that they usually are somewhat... verbose.

Enter stage left: tldr-pages!

tldr-pages is an ongoing effort to create a repository of simplified man pages, that document the most common usages of a command. How about this, for the tar command?

# tar

> Archiving utility.
> Often combined with a compression method, such as gzip or bzip.

- Create an archive from files:

`tar cf {{target.tar}} {{file1 file2 file3}}`

- Create a gzipped archive:

`tar czf {{target.tar.gz}} {{file1 file2 file3}}`

- Extract an archive in a target folder:

`tar xf {{source.tar}} -C {{folder}}`

- Extract a gzipped archive in the current directory:

`tar xzf {{source.tar.gz}}`

- Extract a bzipped archive in the current directory:

`tar xjf {{source.tar.bz2}}`

- Create a compressed archive, using archive suffix to determine the compression program:

`tar caf {{target.tar.xz}} {{file1 file2 file3}}`

- List the contents of a tar file:

`tar tvf {{source.tar}}`

...or this for git reset?

# git reset

> Undo commits or unstage changes, by resetting the current git HEAD to the specified state.
> If a path is passed, it works as "unstage"; if a commit hash or branch is passed, it works as "uncommit".

- Unstage everything:

`git reset`

- Unstage specific file(s):

`git reset {{path/to/file(s)}}`

- Unstage portions of a file:

`git reset -p {{path/to/file}}`

- Undo the last commit, keeping its changes (and any further uncommitted changes) in the filesystem:

`git reset HEAD~`

- Undo the last two commits, adding their changes to the index, i.e. staged for commit:

`git reset --soft HEAD~2`

- Discard any uncommitted changes, staged or not (for only unstaged changes, use `git checkout`):

`git reset --hard`

- Reset the repository to a given commit, discarding committed, staged and uncommitted changes since then:

`git reset --hard {{commit}}`

For those learning linux and the terminal, I think it's an invaluable tool. It helps you out by showing you how to perform common tasks. As you get more experienced though, it becomes useful in another way: showing you how to do those things that you don't do often enough to remember off the top of your head.

I'm probably a bit biased, since I've been contributing to the project for a while (and the nice folks over there recently promoted me to the rank of maintainer :D), so you should check it out for yourself! There's even an online client that you can use without installing anything :-) Once you're ready to install a client directly in your terminal, there's an extensive list of clients documented on the repository wiki, with one available for every environment and platform.

If you encounter a command that hasn't been documented yet, then they've also made it easy to contribute a page yourself.

I think the idea is rather cool, actually - as you've probably guessed by now! Let me know what you think of it in the comments.

How to set up a WebDav share with Nginx

I've just been setting up a WebDav share on a raspberry pi 3 for my local network (long story), and since it was a bit of a pain to set up (and I had to combine a bunch of different tutorials out there to make mine work), I thought I'd share how I did it here.

I'll assume you have a raspberry pi all set up and up-to-date in headless mode, with a ufw for your firewall (if you need help with this, post in the comments below or check out the Raspberry Pi Stack Exchange). To start with, we need to install the nginx-full package:

sudo apt update
sudo apt install  nginx-full

Note that we need the nginx-full package here, because the nginx-extras or just simply nginx packages don't include the required additional webdav support modules. Next, we need to configure Nginx. Nginx's configuration files live at /etc/nginx/nginx.conf, and in /etc/nginx/conf.d. I did something like this for my nginx.conf:

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
    worker_connections 768;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;

    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Not many changes here. Then, I created a file called 0-webdav.conf in the conf.d directory, and this is what I put in it:

server {
    listen 80;
    listen [::]:80;

    server_name plans.helenshydrogen.be;

    auth_basic              realm_name;
    auth_basic_user_file    /etc/nginx/.passwords.list;

    dav_methods     PUT DELETE MKCOL COPY MOVE;
    dav_ext_methods PROPFIND OPTIONS;
    dav_access      user:rw group:rw all:r

    client_body_temp_path   /tmp/nginx/client-bodies;
    client_max_body_size    0;
    create_full_put_path    on;

    root /mnt/hydroplans;
}

Now this is where the magic happens. The dav_access directive tells nginx to allow everyone to read, but only logged in users to write to the share. This isn't actually particularly relevant, because of the auth_basic and auth_basic_user_file directives, which tell nginx to require people to login to the share before they are allowed to access it.

It's also important to note that I found that Windows (10, at least), didn't like the basic authentication - even though Ubuntu's Nautilus accepted it just fine - so I had to comment that bit out :-(

If you do still want authentication (hey! May you'll have better luck than I :P), then you'll need to set up the passwords file. Here's how you create it:


echo -n 'helen:' | sudo tee /etc/nginx/.passwords.list
openssl passwd -apr1 | sudo tee -a /etc/nginx/.passwords.list 
Password:

The above creates a user called helen, and asks you to type a password. If you're adding another user to the file, simply change the first tee to be tee -a to avoid overwriting the first one.

With that all configured, it's time to test the configuration file, and, if we're lucky, restart nginx!


sudo nginx -t
sudo systemctl restart nginx

That's all you should need to do to set up a simple WebDav share. Remember that this is a starting point, and not an ending point - there are a few big holes in the above that you'll need to address, depending on your use case (for example, I haven't included the setup of https / encryption - try letsencrypt for that).

Here are the connection details for the above for a few different clients:

  • Ubuntu / Nautilus: (Go to "Other Locations" and paste this into the "Connect to Server" box) dav://plans.helenshydrogen.be/
  • Windows: (Go to "Map Network Drive" and paste this in) http://plans.helenshydrogen.be/

Did this work for you? Have any problems? Got instructions for a WebDav client not listed here? Let me know in the comments!

Running Prolog on Linux

The learning prolog banner. Hello! I hope you had a nice restful Easter. I've been a bit busy this last 6 months, but I've got a holiday at the moment, and I've just received a lovely email about my learning prolog series. It's reminded me about how much I actually rather like (swi) prolog (I'm a bit nuts, I know :P), and I've decided that it would be splendid if I could get back into it a bit.

I'm not sure whether it'll go anywhere, but I'm going to look into web crawling and parsing HTML. Anyway, since I'm re-discovering prolog, I had a teensy bit of trouble getting prolog to run one of my old scripts just now, so I thought I'd blog about it to save me the trouble next time :D

Running prolog scripts on linux isn't actually that hard. First, you need to install SWI-Prolog:

sudo add-apt-repository ppa:swi-prolog/stable
sudo apt install swi-prolog-nox

Then, you run a prolog script like this:

swipl -f ./Awesome-Prolog.pl
Welcome to SWI-Prolog (threaded, 64 bits, version 7.4.1)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit http://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- 

...then you can interact with your Prolog knowledge bases as normal. When you want to exit, the easiest way I've found is if you press CTRL + D.

If, however, you're lazy and don't want to type prolog -f AwesomeSauce.pl out every time you want to run your script, you can prefix your file with the following shebang:

#!/usr/bin/prolog -f 

Make sure it's the very first line in the file. Prolog appears to ignore it and carry on regardless. Then you can run it like this (making sure you've chmod +x'd it first:

./CoolBot.pl
Art by Mythdael