Own your code, part 3: Shell scripting infrastructure
In the last post, I told the curious tale of my unreliable webhook. In the post before that, I talked about my Gitea-powered git server and how I set it up. In this one, we're going to back up a bit and look at setting up Laminar CI.
Laminar CI is a continuous integration program that takes a decidedly different approach to the one you see in solutions like GitLab CI and Travis. It takes a much more minimal approach, instead preferring to provide you with the tools you need and letting you get on with setting it up however you like and integrating it with whatever you like.
laminarc queue build-code laminarc show-jobs
It is, of course, entirely command-line based. In order to integrate it with other services, webhooks are needed. In my case, I've used webhook for this purpose. Before we get into that though, we should outline how the system we build should work.
For me, I'm not happy with filling a folder with job scripts. I want my CI system to have the following properties:
- I want to have all the configuration files and scripts under version control
- I want to keep project-specific CI scripts in their appropriate repositories
- I don't want to have to alter the CI configuration every time I start a new project.
- The CI system should be stable - it shouldn't fall over if multiple jobs for the same repository are running at the same time.
Bold claims. Achieving this was actually quite complicated, and demanded a pretty sophistic infrastructure that's comprised of multiple independent shell scripts. It's best explained with a diagram:
There are 3 different machines at play here.
- The local computer, where we write our code
- The git server, where the code is stored
- The CI Server, which runs continuous integration tasks
Somehow, we need to notify the CI server that it needs to do something when new commits end up at the git server. We can achieve this with a git post-receive hook, which is basically a shell script (yep, we'll be seeing a lot of those) that can perform some logic on the server just after a push is complete, but the client pushing them hasn't disconnected yet.
In this post-receive hook we need to trigger the webhook that notifies the CI server that there are new commits for it to test. GitHub makes this easy, as it provides a webhook system where you can configure a webhook via a GUI - but it doesn't let you set the webhook script directly as far as I know.
Alternatively, should we run into issues with the webhook and we have control over the git server, we can trigger the CI build directly by writing a post-receive git hook directly utilising SSH port forwarding. This is what I did in the end for my personal git server, though as I noted in part 2 I did end up working around the webhook issues so that I could have it work with GitHub too.
For the webhook to work, we'll need a receiving script that will parse the JSON body of the webhook itself, and queue the laminar job.
In order for the laminar job to work without modification when we add a new project, it will have to come in 2 parts. The first will have a generic 'virtual' or 'smart' job, which should create a symbolic link to itself under the name of the repository that we want to run CI tasks for.
When called by Laminar under a repository-specific name, we want to run the CI tasks - but only on a copy of the main repository. Additionally, we don't want to re-clone the repository each time - this is slow and wastes bandwidth.
Finally, we need a unified standard for defining CI tasks in our repositories for which we want to enable continuous integration.
This we can achieve with the use of the lantern build engine, which I'll talk about (and its history!) in a future post.
Putting all this together has been quite the the undertaking - hence this series of blog posts! For now, since this blog post somehow seems to be getting rather long already and we've laid down the foundations of quite the complicated system, I think I'll leave this post here for now.
In the next post, we'll look at building the core of the system: The main laminar CI job that will organise the execution of project-specific CI tasks.