The Rewrite

[roadrunnertwice]

I run Eardogger.com, the web's favorite unpopular bookmarking tool for binge-reading webcomics. It's about four years old, and it's currently built on Typescript, Node.js, the Express web framework (with some add-ons), and PostgreSQL.

I've been rewriting the whole thing in a new Pile of Stuff — Rust, the Axum web framework (and the Tokio + Tower + Hyper ecosystem it's built on), Sqlite, and my experimental FastCGI/HTTP reverse-compatibility layer. (Yup: this was the secret endgame for that whole project.)

All features are complete. I've got the rewrite deployed in realistic hosting to do some soak testing, and it's working amazingly well. Most of the remaining to-dos are either rewriting my integration tests, or ancillary stuff like backup jobs, release scripting, and data import scripting.

A rewrite, huh? 😒

A total rewrite of an existing piece of software is traditionally considered either a very distressed move... or else a very distracted move, from someone who can't leave well-enough alone.

I can't fully deny the latter, tbh! Eardogger's had a longstanding role as a project I unnecessarily fuck around with when I'm in the mood to learn something new, and I did have a couple tools I wanted to get familiar with prior to having my next weird web idea.

But, okay, listen: there's a little bit of distress there too. The last time I messed around with Eardogger was because the terms of the site's hosting changed radically in late 2022 and I had to bust my ass on someone else's schedule to move platforms.

And I learned some things from that experience!

• A Postgres server is a 📺 seabird necklace. It doubles the services I need to host and maintain! My current architecture actually has app and db on completely separate platforms, which is extra berserk. (Also, Postgres major version upgrades are harrowing; one upside to my forced migration was that it happened before I hit a forced upgrade.)
• The build/maintenance/runtime characteristics of a Node.js app constrain my hosting options more than I originally thought they did.
• "The cloud" (i.e. collecting rent on highly granular computing resources) is hella unstable if you're a scrub. You can buy stability if cost is no object, but otherwise you're a nomad and should plan accordingly.

A list of demands

I mean to keep Eardogger running indefinitely. AND I'm lazy and distracted.

Here's what I think a lazy and distracted person needs in order to keep a public service online for decades:

• Stability: the freedom to leave it be, running without fuss, for many years at a time between bouts of inspiration.
• Flexibility: if I do have to move platforms, my options should be as unconstrained as possible, and the move itself should be as painless as possible. Travel light.
• Affordability: honestly I'm fine paying for hosting (even though I've run Eardogger on free tiers for its whole life so far 😅), but the cost should be predictable and it shouldn't be totally disproportionate to the benefit it provides. (Heroku wanted like fifteen bucks a month, which is ludicrous.)

That doesn't feel like too much to ask! Mostly just because this app doesn't DO a whole hell of a lot. But after exploring my options, I decided my current stack couldn't deliver it.

So the new stack will get you that plus a pony, right?

Uhhhhh, we'll have to wait and see! The async Rust ecosystem is not totally mature yet, Sqlite is still widely considered a questionable choice for server-side use, and absolutely nobody thinks that "busriders" shit is a prudent move. So there's a reasonable point of view that says I'm being a dumbass.

But I feel good about my analysis here.

First, Rust is a compiled language that doesn't rely on a thick external runtime, so once you've built a binary, it keeps working as long as you've got an OS that can run it. Part of the ongoing problem with a Node.js app is that you (and all your dependencies) have to keep up with the versions of Node that can run in your hosting environment.

Also, it's fast. That's usually irrelevant to a web app, but a fast startup time is a game changer. There's several hosting scenarios that shut your app down after a while and re-start it on demand, and you cannot use those if you take several seconds to start up. (I've tried with the current Eardogger, and it was Bad.) But my soak test with the rewrite runs like that, and it's starting up from cold sleep all the time, and you can't even tell. So, sometimes speed can buy you more options.

As for Sqlite: it has some real limitations. It takes more effort to write a performant server-side app, because you have to take on a lot more of the responsibility for managing concurrent access; this pretty much rules out any kind of ORM, so you're definitely doing your data layer by hand. (This article gives a very good rundown of the practical considerations.) It also constrains your ability to scale; multiple app servers can't easily share a database, so you'd better be able to handle all your traffic with a single instance.

But! Everything is a trade-off, and what you get in exchange for all that is massive:

• No database server.
• Really fast data access.
• ✨ No incompatible database upgrades. ✨ The maintainers have committed to keeping the format forward-compatible through like 2050 or something, which is wild to think about for anyone who's had to rebuild their Postgres database from scratch during an upgrade.

If you've got low traffic and an expected service lifetime of decades, that's, uh... persuasive. To say the least.

And as for busriders and FastCGI... time will tell, but I'll say two things. First, FastCGI itself ain't going anywhere, because it's how everyone keeps PHP out of their webserver, and PHP is ineradicable (and must be kept out of your webserver process at all costs). Second, my entire design for busriders was all about hedging my bets, so the rewrite is a normal HTTP app under the hood. If this "smuggle modern app into shared hosting" scheme doesn't work out long-term, I can go practically anywhere that'll give me a fraction of an OS, a port, and an attachable storage volume. My bags are packed and I travel light.

Also it was fun, right?

Haha yeah, it totally was.

Axum is really nice to work with! It combines some of my favorite parts of Express (Eardogger's old framework) and Bevy (the game engine I've been working with on another project). And the way it's able to give strong type guarantees to handlers (despite being flexible and comfy to work with) means the code can inherently represent a bunch of assumptions that otherwise I would have to remember somehow; that should be really nice for future maintenance after I haven't touched it in two years.

I'm doing the database stuff in sqlx, which is mostly pretty great. I'm using the compile-time-checked query macros; it's true, they add some significant extra build weirdness, but once again, it's all about the maintenance — those automatic checks let me skip a huge amount of otherwise-manual testing. During development, it caught a ton of bugs and schema problems before they could even take root.

For templates I'm using minijinja, and it's all right. I've never met a template language I love yet, so it's par for the course. I investigated a lot of alternatives, and I think the design constraints of this one seemed easiest to live with.

And using busriders makes it feel like I'm somehow beating the system. 😛 I love it.

Here's the code for the rewrite, if anyone's interested.

Comments

[sporky_rat]

Eardogger love!

[roadrunnertwice]

🌻

[kaberett]

(This was a delight to read; thank you for writing it up!)

[roadrunnertwice]

Thank you!