Busride-rs

[roadrunnertwice]

Okay, so I know this is going to shock you, but I've been working on something arcane and impractical.

I'm in a new band called Kuwait Grips

I made a wrapper for normal HTTP-speaking Rust web apps so their traffic can take an extra round trip through a totally different protocol, before being translated back into HTTP for the outside world. Specifically, I plan to serve Axum-based apps via FastCGI, a protocol that went out of fashion in the mid '00s.

This probably sounds dubiously useful, but, man, listen,

The Fallen Age

Your experience might differ, but here's how I see it: over the last decade-plus, it got a lot easier and cheaper to run professional-scale web apps, and a lot harder and more expensive to run amateur-scale web apps.

Then

The way you used to run a web app was, you get some cheap-ass shared web hosting, you unzip a honk of PHP files into a directory and do some setup, and then you have a Wordpress or whatever. Someone else is responsible for keeping the computer online, and you just need to manage your app.

Now

How things work nowadays is, you gain control of a computer somewhere, and you start a program on it that runs forever and takes sole ownership of a system-wide resource. You're responsible for keeping both the computer and the program working.

Modern web frameworks in Rust, Go, and server-side JavaScript all expect to work like this... and old-style shared web hosting providers don't allow it. So to run a modern web app, you have to get a whole computer to yourself.

Mind, that's a lot easier and lighter-weight than it ever used to be; you can use things like containers, PaaSes, or just a little VM on DigitalOcean or somewhere.

But "easier" ain't "actually easy!!!" It's a long-ass number of miles away from there.

As it happens, my day job involves building stuff to make it easier to manage cloud infrastructure, and I can confidently state that our best tools will never rival "unzip some PHP files and let someone else sysadmin for you."

I Just Want What the Ancients Had

I maintain at least one little web service, and it's likely I'll end up with more. This kind of service is niche and low-traffic; it doesn't need max performance or dedicated infrastructure.

I also already have some old-school shared web hosting on DreamHost that I use for my dad's website, and adding new sites to it doesn't really cost any extra.

I would like to just put my little low-traffic services on my shared hosting! You know, like we used to. My apps do not need to reserve a whole-ass towncar, they would be perfectly happy just riding the bus.

Is there any way that I can write little apps using modern, nice-to-develop-in web frameworks (all of which generally expect to run their own server process), but host them on an affordable shared hosting account that won't allow me to run my own separate web server? Can't I just live the PHP lifestyle, without having to write PHP?

The Lost City

What if I told you there was already a way to run your own arbitrary server processes on shared hosting, while staying totally within the bounds of what your host expects? Your app server can't permanently daemonize, but in return, the web server is willing to manage its entire lifecycle for you; it'll spin up a process on demand, proxy traffic to it, keep it running for as long as it seems busy and interested, kill it to save resources if there's no traffic for a while, and bring it back as needed ad infinitum.

For a low-traffic service, this seems practically too good to be true: it's an efficient use of resources (especially when your app's napping), AND an efficient use of your limited human bandwidth for babysitting network services; you don't even need to fret about keeping your process alive, you can just install and go!

There's only one catch: your app server can't speak HTTP. 🥲

Yup, that's right, I'm talking about FastCGI. I'll try to avoid going on a whole deep dive about this, and just keep it to the bullet points:

• "It's just a faster version of CGI" factoid is 80% untrue. In fact, FastCGI is a client/server network protocol with its own binary message format, designed to encapsulate HTTP requests and responses so that a web server can delegate requests to a separate application server.
  • There were several libraries that let you use it kind of like CGI. It also has "authorizer" and "filter" roles that muddy the waters even further. But our goal is to completely ignore the intended application programming model here, so let's move on.
• The main remaining use of FastCGI is for PHP. Nearly everyone doing PHP in prod runs PHP-FPM at a stable address, then points Nginx or Caddy or Apache's mod_proxy_fcgi at it.
  • This "proxy to known address" paradigm won't work for us, because the whole point is that we can't daemonize and claim a permanent port or socket.
• However, the original FastCGI spec also defined a scheme like what I described above, where the web server manages the lifecycles of any number of app servers. Apache's optional mod_fcgid add-on works exactly like this... and DreamHost has it enabled by default!

If You Want to Ride the Bus, You Gotta Take a Transfer

So to sum up, I want to write modern, normal HTTP apps... but if I were willing to speak the FastCGI protocol instead, then I could have the low-touch deployment mode that I want.

Why not shim in a translation layer that converts FastCGI requests to HTTP requests, forwards them to the application code, and translates the responses back to FastCGI? Yeah, it's a bunch of extra computation that's technically pointless, but it'll work today on the hosting I already have.

And just as importantly, it'll protect me from having to architect the app itself around FastCGI. I can actually compile a single binary that can either ride the bus on shared hosting, OR run standalone in a container or VM. So my forward-compatibility story is safe, if this approach stops working or if an app gets too popular and needs more elbow room.

Recipe for Sky Pie

Obviously this whole weird scheme is pretty non-optimal. What would be much nicer would be a forward-thinking, low-overhead way to efficiently manage processes and divide up shared server resources that was designed around modern apps that speak native HTTP.

Such a thing is not forthcoming.

Corporate users have all the money to devote to infrastructure development, and the superior performance and workload segmentation guarantees of modern cloud infrastructure are a much better fit for their needs. Accordingly, the closest relatives of what I'd want are all based on containers or firecracker VMs with worse wake-from-nap latency and worse end-user complexity, or else on serverless frameworks and edge functions that you have to architect your whole app around.

No one's ponying up the money to build me a better bus... so it's off to the junkyard we go, with our wheelbarrow full of welding equipment!

Anyway, I Did It!

Here's a little 3m demo I recorded when I got my initial proof-of-concept working. If you know anything about deploying a self-hosted app in the 2020s, it will shock and scandalize you.

Watch on YouTube

And, here's the code itself, including a demo project:

• nfagerlund/busride-rs

I found a FastCGI server library for Rust (I'm SO curious about why the author made this, but yeah it's very precisely what I needed) and put together a server loop that translates between the normal HTTP that an inner app understands and the FastCGI protocol that Apache is willing to accept. As long the binary you build knows how to start up in the weird environment that classic FastCGI provides, you can just install it, drop in an .htaccess file, and wander off to go do something else.

At the moment, it's Axum-specific and has to be built into your app as an alternate server mode. In theory it ought to be possible to make a fully generalized wrapper that can spawn any program as a child process and proxy real-actual HTTP to it, but that's more work than I want to do on this; at the moment, this should work fine for me.

So... Why??

Here's another interesting point about apps that run in this mode: anyone else can install them on their shared hosting just as easily, if I give them a build and a README.

In the last few years, there's been a medium amount of big talk about how we need to re-wild the interwebs; bring back some spirit of curiosity and generosity and chaos that we thought we perceived in the '90s and the '00s.

In a recent thread that rolled across my Mastodon feed (wish I could remember and link it, but it took a while to percolate before I took it to heart), someone pointed out the short version of what I described above — that hosting has gotten better for pros at the expense of amateurs — and then said: if we think there's a connection between self-hosting and re-wilding the web, then we're going to have to reverse that, because getting out of a tech-dominated world of walled gardens is going to require empowering the type of normal users who could kinda-sorta keep a Wordpress installation afloat back in the day but who have no hope of, say, sysadmining a Mastodon instance.

I've been thinking about that in the background, a bit.

Comments

[sporky_rat]

YAY!