PhD Update 16: Realising the possibilities of the past

Hey there! It's been a while. As I explained in a previous post, I've been adjusting to a new part-time position in my department! In short, posts will continue on here by may be slightly less frequent than before.

Before we begin, here is the customary list of previous posts in this series:

• PhD Update 1: Directions
• PhD Update 2: The experiment, the data, and the supercomputers
• PhD Update 3: Simulating simulations with some success
• PhD Update 4: Ginormous Data
• PhD Update 5: Hyper optimisation and frustration
• PhD Update 6: The road ahead
• PhD Update 7: Just out of reach
• PhD Update 8: Eggs in Baskets
• PhD Update 9: Results?
• PhD Update 10: Sharing with the world
• PhD Update 11: Answers to our questions
• PhD Update 12: Is it enough?
• PhD Update 13: A half complete
• PhD Update 14: An old enemy
• PhD Update 15: Finding what works when

A lot has happened since last time! I'm going to split this up into sections as I have in previous posts, but in summary the noodling around I've done since the last post has really paid off.

Publication

After another round of revisions, my journal article on my social media research has now been accepted and published!

View it here:

https://doi.org/10.1016/j.cageo.2023.105405

It's my first published journal article, so I am quite excited about it :D

If I haven't already (I'm writing this post first, as I only got notified about its publication while I was writing this post!), I'll definitely be making another post about it here!

Rainfall Radar

The main thing I've focused on a lot is my rainfall radar model, and beating it into some kind fo shape that actually works. This has not been a simple process, but I think that this graph speaks for itself:

It works! I can scarcely believe that after nearly 3 and a half years it finally produces a useful output. This feels like a big personal achievement - as those who have been following this series will know, I have tried many different things before reaching this point.

The first question I know will be on your mind is "How did we get here?", and the answer to this lies in 2 things:

1. Connectedness
2. Resolution and boundary difficulties

Let's tackle connectedness first. By connectedness, I mean specifically parts of a given model connecting back on themselves. This is important for multiple reasons, not least because it reduces the effect of the vanishing gradient problem. This is also the reason that ResNet adds skip-connections, as then the gradient weight updated used when the model is backpropagating can flow all the way up to the top of the model. Without this, the weights in the initial layers can't update, and information is then lost before it even makes it very far into the model.

From what I can tell, this is the primary problem with the models that I have tried so far, in one way or another. Autoencoders, for example, do not have very much connectedness, making it difficult for backpropagation to do its thing.... especially when the task at hand is a significantly difficult one.

The solution I ended up employing here was DeepLabV3+. It uses an image encoder at first, and then a PSPNet-style pyramid scheme for analysing multiple scales of features, and then finally an image segmentation head. It also has a skip connection between halfway up the image encoder and the segmentation head too, further increasing the connectedness of the model.

Once I had something from the initial DeepLabV3+ model, improving the output was a matter of increasing the resolution of the output and adjusting things so that the model can better resolve the boundaries of the water / no water regions I was asking it to predict.

To this end, with this tweaking I describe DeepLabV3+ has turned out to be the ideal model for the task. The changes and hyperparameters I used can be summarised like so:

• Loss function: Add dice loss to cross-entropy loss.
• Learning rate: Reduce to 0.00001 from the default 0.001
• Upscaling: Hack the model to upscale the input/outputs. This increases the resolution the model operates at, improving performance significantly
• Removing isolated pixels: I removed water pixels with no neighbour pixels being water. This is like a band-aid on the real problem (a bad physics-based model run), but it did help.

With this working models, I can now consider the other avenues I was exploring in part 15 of this series reasonably as dead ends, though I have learned a lot by investigating them.

I consider this model to be a proof of concept only. The idea needs a lot more adjusting and improving before it will be actually useful to anyone. Still, it might be improve short-medium term (i.e. ~up to a few hours in advance) flooding forecasts with some a lot more work I think. While my focus currently is writing up my thesis (see below), I do plan on continuing to work on this and other research projects on the side on a long-term basis. One of my many goals is to wrangle this model into something more than just a proof of concept, and somehow measure it's effectiveness more precisely.

The long road to thesis and beyond

With my funding for the main research period of my PhD at an end, my focus has been shifting to the writing of my thesis. The feeling is actually quite surreal - for the longest time writing my thesis has been a mystical objective far off in the distance in a blurry haze, but now the details are very much resolving into something more tangible.

So far I have a draft chapter based on my recent journal article, and part of a chapter on the rainfall radar model I've talked about briefly above. I also have part of an introduction and a background sections, but these require significant reworking because I wrote them ages ago (they are just bad).

The plan is to have my thesis complete by December 2023, potentially giving the required 3 months submission notice in ~September 2023 - depending on how things go with writing.

When my PhD comes to a close, that will also mean the end of this series of blog posts. I think this is the longest series of blog posts I've ever posted here, and certainly one of the most personal. This does not mean the end of posts on here about my research though, as I plan to continue blogging about it on here. The form this will take is likely to be similar to the form that the posts for my PhD have taken.

As I don't currently know what form my research will take after my PhD, I cannot say what will happen about blogging about it, only that it will happen ;-)

Thanks for sticking with me throughout this long and at times difficult process - it's been a wonderful and wild ride! Even as this part of my journey is beginning to come to a close, I really appreciate all the help and support everyone has given me throughout the process.

I'll try my best to keep up with this series again, now that I've had some time to adjust to being experimental officer. Until next time!