Weeknote: 2024-W26-27

08 July 2024

I completely forgot to do a weeknote last week, so here’s a double issue, covering two weeks of work around file management, mostly around uploading and storage.

Uploading

I’ve been wanting to improve the uploader for ages; originally it was written by an open source contributor (which was superb), and I’d never really looked at it in any detail, so it was time to give it some love.

I’d actually been working on underlying infrastructure for file handling changes for ages; rather than just managing files directly, a few weeks I’d added in another layer, a system called Shrine. This was a huge refactoring under the surface, and honestly it was pretty scary! Changing how all the files are managed, in a system where that’s the primary job, and where it will be deployed to a whole load of users without me seeing? Terrifying.

Anyway, I’d written the low-level stuff a few weeks ago, in late May, but it wasn’t finished; for one thing, all the tests were failing. So, first job for the week was to get those passing, then manual testing to make sure that all the new file management was working properly. I got that merged in an deployed on my own system to start building my confidence in it at the start of the week.

Then it was time to put all that work to good use, starting with the uploader. Shrine integrates nicely with a Javascript uploader called Uppy, and between them they give me a lot of options. I’ve started out with just a straight rewrite of the upload page using the Uppy dashboard, but in future I can do a lot more with it, like being able to add files by URL, which I really want to be able to do.

Background jobs

A side effect of this rewrite was to change the background jobs around a bit. So that long-running high performance jobs don’t clog up the rest of the app, I’ve moved them to their own queue with its own dedicated worker. That means that things like file scans will continue to happen quickly even if that’s doing a long geometric analysis job; and it also means you can scale the workers separately for the different job types.

The uploader rewrite meant also that we could now do more intelligent creation of models during upload; instead of just putting things on disk and doing a full rescan, we can create the models and attach files as we go, making it much much quicker to work with new models after upload.

Satisfyingly, too, I made the uploader accept individual 3d files to upload as a model, so you don’t have to zip things up any more if there’s just one file. This was something a lot of people asked for, so it was great to be able to quickly get it in there.

All aboard the release train

So, all this got pushed out in the very scary v0.70.0 release, which for good measure also included a bunch of underlying software updates like Rails 7.1 and Ruby 3.3.1, which we’d been blocked on for a while.

That version was then followed rapidly by v0.70.1, v0.70.2 and v0.70.3, as users took to the support channel and reported various issues which I got to fixing. I was expecting issues for this release, so it wasn’t unexpected; but I was surprised that the actual file handling wasn’t the problem in all these cases. One was because I forgot to merge a branch that sped up the data migration, a few startup problems, and one for people with lots of libraries. So, all that underlying work I’d been so scared to put out there was… OK!

Cloud storage

Once that release settled, it was time to move on to the big reason for adding Shrine; the ability to use completely different storage services underneath. The Shrine rewrite had been done with just one storage service, the local filesystem, but now it was time to add another; Amazon’s S3 object storage. That means that instead of having your library on local disk, you could put it all out onto the infinitely expandable storage of the Internet at large. S3’s API is the de facto standard for cloud object storage, and there are loads of commercial and open source self hosted systems that provide it; so, by adding S3 support, we give people a lot of hosting options.

The idea was that now we have Shrine mediating the file management, we should be able to just swap out the backend. Refactoring is one thing, but the moment you put it into action is another. You do all this work to make the change easy to make, and then…

… it works? It actually worked, and it was pretty easy. All the groundwork paid off, and straight away I could upload files and extract them straight to S3 without any code changes. Incredible.

There were a few bits of direct file access still straggling that I hadn’t caught in the initial refactor, so I tidied those up, but then at the end of week 27, I pushed out v0.71.0, with cloud storage support and big improvements to the library creation/editing UI.

The S3 stuff itself took me two days, but then I spent the next two days trying to make some Javascript behave on the library editing page before I could release.

Trust me, if you enjoy hard things being easy, and easy things being hard, you’ll love software engineering.

Next up

With cloud storage done, that is Milestone 1 (Core Feature development) of the NLNet / NGI Zero funding complete! I’ve got a little bit more work on the deployment, documentation and compliance milestones, but then the remaining funding is all in the ActivityPub and streaming 3d side of things, which I’m looking forward to getting into.

I think this week I’ll try to tick off a load of those remaining bits, particularly coming up with a simpler application setup which doesn’t need separate containers for the database, and hopefully a linuxserver.io container. Then we might well be ready to take the big step to v1.0.0!