Roadrunner Twice

As mentioned previously, this year I switched to hosting eardogger.com in what's either a highly unconventional environment or an unusually conventional environment, depending on your perspective. This has mostly gone completely fine! However, I did have one incident several weeks ago, and it was a funny one.

I was out reading webcomics on my phone, and got creepy 500 errors on Eardogger; when I got home, the logs showed a Resource temporarily unavailable error when trying to access the database.

⁉️ (Metal Gear Solid guard alert noise)

All right, first off: That database isn't a remote server; it's a file on the local disk. If THAT's "unavailable," something's very wrong. A quick web search indicated that error comes from the operating system itself, not anything in my tech stack (like sqlite maybe). At some point, I visited a page on the site, then tried to run a command in my SSH session:

$ ls
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

Hahahaha holy shit.

Ok anyway, long story short: I was hitting my user's process limit and being prevented from spawning new processes or threads. The 500s were happening when concurrent DB reads would have made the reader pool spawn a new thread, and it hit the wall instead.

A given user may only run a certain number of processes at once on this server. Eardogger is at maximum a single process instance, so I thought I was fine. But then my web host upgraded my server's OS, which changed the process limit accounting to also include sub-process threads. And Eardogger IS multi-threaded.

How multi-threaded, exactly? Well, I was using the Tokio multi-threaded runtime with default configuration. And it turns out the default behavior is to immediately spawn one worker thread per logical CPU core...

...on what turns out to be a 128-core web server. The process limit (not advertised, but support will divulge if you ask) is 25.

I wouldn't want that even if it WAS allowed!! This app has like three goddamn users! I made my thread pool configurable and set it to single digits, and that immediately banished the errors and the shell lockups. 🌈 As a bonus, it also cut the app's cold startup time from "barely perceptible" to "legit gone" — apparently spawning more than a hundred threads on startup takes a noticeable amount of time, but Rust is so fast in general that it covered most of that sin and I wasn't immediately suspicious.

Lessons learned:

• Resource heuristics that inspect the local system's capacity are a huge red flag, because they act normal on your laptop and then go berserk elsewhere. Set your ceiling explicitly!
• top and ps on Linux don't list threads by default, you have to use an extra argument to see that.
• Wow.

EDIT: But actually, in most modern deployment scenarios, your server's CPU and memory resources really are much closer to the scene on your laptop, since the standard practice is to slice up computer resources into tiny single-purpose shards via containers or VMs. Like, Ruth was showing me something from her work that we would both consider "fairly extreme" in terms of resource allocation, and I was like "oh yeah, that's a whole lot of laptop... but it ain't a web server, you know?" So yes, this whole problem did in fact stem directly from my runtime environment being weirdly atavistic, and most people are probably fine with Tokio's default behavior.

I run Eardogger.com, the web's favorite unpopular bookmarking tool for binge-reading webcomics. It's about four years old, and it's currently built on Typescript, Node.js, the Express web framework (with some add-ons), and PostgreSQL.

I've been rewriting the whole thing in a new Pile of Stuff — Rust, the Axum web framework (and the Tokio + Tower + Hyper ecosystem it's built on), Sqlite, and my experimental FastCGI/HTTP reverse-compatibility layer. (Yup: this was the secret endgame for that whole project.)

All features are complete. I've got the rewrite deployed in realistic hosting to do some soak testing, and it's working amazingly well. Most of the remaining to-dos are either rewriting my integration tests, or ancillary stuff like backup jobs, release scripting, and data import scripting.

A rewrite, huh? 😒

( yeah dude )

Also it was fun, right?

Haha yeah, it totally was.

Axum is really nice to work with! It combines some of my favorite parts of Express (Eardogger's old framework) and Bevy (the game engine I've been working with on another project). And the way it's able to give strong type guarantees to handlers (despite being flexible and comfy to work with) means the code can inherently represent a bunch of assumptions that otherwise I would have to remember somehow; that should be really nice for future maintenance after I haven't touched it in two years.

I'm doing the database stuff in sqlx, which is mostly pretty great. I'm using the compile-time-checked query macros; it's true, they add some significant extra build weirdness, but once again, it's all about the maintenance — those automatic checks let me skip a huge amount of otherwise-manual testing. During development, it caught a ton of bugs and schema problems before they could even take root.

For templates I'm using minijinja, and it's all right. I've never met a template language I love yet, so it's par for the course. I investigated a lot of alternatives, and I think the design constraints of this one seemed easiest to live with.

And using busriders makes it feel like I'm somehow beating the system. 😛 I love it.

Here's the code for the rewrite, if anyone's interested.

Julia Evans’ StrangeLoop 2023 keynote was about digging into the different reasons a tool can be hard to learn, and it was a real good talk! At the end, as a tossed-off addendum to a conclusion about continuing to learn things, she said “I still don’t know why Git is hard.”

I happen to have thoughts about that one!

I have had to teach a fair number of (generally clever and persistent) people how to get around in Git or how to use its more advanced features. While doing so, I have often failed to get the basics to stick, which is incredibly aggravating to someone who prides themselves on explaining things. Sometimes this devolves into me talking about wave/particle duality as a crucial metaphor for getting through a rebase intact, and everyone in the room looking at me like my second head just tried to convert them to Gnosticism.

So, I’ve spent some time thinking about this before this weekend.

( Read more... )

I just finished doing a major update to Eardogger.com, my simple little bookmarks-for-webcomics app! 🙌🏼

What’s changed? Uhhhhh almost exactly nothing.

• If you were getting round-tripped to the “saved!” page and back on mobile Safari instead of the nice unobtrusive “ok done” banner, it’s fixed! Generate a new bookmarklet to get the benefit.
• The delete buttons now have a (hilarious, IMO, but I’m easily amused) spinner effect while they’re working, though blink and you’ll miss it.
• If you have more than 50 dogears, they’ll get split into multiple pages. No one will ever see this. 😹 But pagination of unlimited data sets is good practice, since it makes you a little harder to DoS.

Yep: pretty much all the work was internal updates with very little visible effect. And boy, there were a lot of them. But hey: Eardogger basically exists in the first place for me to tinker with and learn things. I made it because I wanted it to exist, but I made it the way I did because I wanted to pick up some new skills.

So here’s what I got to monkey with this time:

( Excessive code exposition )

Guess what, I launched a web app last week! Almost two weeks to the day after I realized I was probably capable of doing it, which, huh, wow. Anyway, I sorta know node.js and SQL now.

SO, introducing Eardogger.com. It's a bookmarking service for gradually reading through serial archives, like long-running webcomics or other online stories. This isn't a wholly original idea, but Eardogger has a singular advantage over prior art: it's incredibly fucking stupid. (I know that sounds like self-deprecation but it's actually an outrageous brag.) It gives you a context-sensitive pause/resume button that works across all your devices, and then it basically stays the hell out of your way.

Free to use, and sign-ups are open; take it for a spin and start catching up on some stories you've been meaning to get to.

Well, and if I'm gonna shill a thing for reading webcomics, I should probably also tell people what's good. Here's eleven things I think are rad which have at least medium-hogwild backlogs.

( in alphabetical order )

It probably goes without saying, but this is some classic ADHD technology. There's a handful of webcomics I've been meaning to catch up on for years, and I never got around to it because manually keeping track of my place was too hard.

Like, it literally isn't!! Normal people manage just fine. (Well, either that or they don't read webcomics in the first place, but what kinda life is that.) But if I'm aware that something is pointless busywork, trying to make myself do it is basically the fucking apocalypse, such that learning two new programming environments over a couple weeks honestly seems easier. IDEK.

Regardless, it's been a fun project. I'm planning to eventually add a legit browser extension and maybe an iOS share sheet extension, but the existing bookmarklet interface is working well enough that tbh I'm more interested in reading some comics right now.

I think I’ve finally wrapped my head around promise-based async logic in JavaScript. It took a while!

To celebrate, here’s the analogy I wish someone had given me ages ago:

A promise is a black hole. Once a value crosses the event horizon into a promise, it can never come back out again to interact with synchronous code. It’s in another universe now! You can send values from synchronous code in there to interact with the promise universe, but they’ll also have to stay there once they cross the event horizon. And it turns out that this is all fine and doesn’t limit you very much at all; it’s just that when you’re in the promise universe, any logic has to be of the form “do this once this condition is met,” never just plain “do this.” Different physics inside the black hole.

Anyway, once I started thinking of it like that, it all seemed perfectly reasonable. It also clarified some things that seemed arbitrary before, like why you can’t call await unless you’re in an async function. (It’s because await is just a clever balancing of terms so you can think synchronously while still obeying async physics.) I gather there ARE other languages that have promises where you can be like “no really, block the main thread and wait for this to resolve,” but since JS’s whole design philosophy treats blocking as such an apocalyptic event, it makes sense that you can never come back from async.

OK, so, bookmarks.

Web browsers have had bookmarks for more than 25 years, and we're basically accustomed to how they work. Nowadays they sync across your devices, and there are some oddball extended implementations out there (like Pinboard, or Zotero), but they still mostly just act like good ol' bookmarks.

Except, hold on — do they act like bookmarks? Where are the holes in that metaphor? What is a bookmark?

Out in wood-space,* there are multiple kinds of bookmarks. Here are the ones I can think of:

• Cursor bookmarks, which move to track your (mostly linear) progress through a text so that you can pick up where you left off.
• Absolute bookmarks, for remembering a specific location for later reference. These are a research tool, usually implemented as color-coded post-it strips, and you usually use many of them at once.
• Hypertext bookmarks, which make it easier to jump between a main text and its endnotes. These are basically just additional cursor bookmarks that track a parallel text stream.

Web bookmarks are absolute bookmarks, and IMO they're even better at that task than their namesake. (Well, except for the fact that the web is shifting and impermanent and links eventually rot. But never mind that for now.) And hypertext bookmarks are just a crap workaround for a lack of hyperlinks (ugh, endnotes), so the web was always one-up on that.

But if someone refers to a bookmark in a wood-space context, 90% of the time they're talking about a cursor, and browsers are crap at cursors. The only real native equivalent is when you leave a tab open for months. You can sort of re-implement cursors with absolute bookmarks, but what you're really doing is taking out a new post-it strip to mark your current spot and then, as a separate operation, yanking the last one and throwing it away, which just feels like way too much effort when you know in your heart that you just want to move your cursor. It's all enough of a pain in the ass that it's deterred me from catching up on a bunch of webcomics that I legitimately want to read.

I only know of a few efforts to address this over the years. A few serials have rolled their own cookie-based "mark my spot" features, but those have generally been local-only, which maybe made sense when you had exactly one computer and is now mostly useless. And then there was ComicRocket/Serialist: that was a respectable attempt at a general solution, but it relied on a cached database of post order for each supported site, and now that whole edifice has mostly rotted out and it's not really usable anymore. IMO their goal of an integrated back/forward nav was over-ambitious, and sabotaged the really crucial cursor part of the project.

After thinking about this for a few days, I'm convinced that it's possible to solve about 70% of the problem (as I see it) with an honestly very stupid webservice and a bookmarklet. So in my downtime, I'm dinking around on glitch.com to see if I can get a prototype up and running.

* It's not meat-space because we mostly stopped making books out of meat. (Vellum may have its good points, but it's so expensive and heavy.)

The wild compounding blowout one! That’s the one that looks like this:

You can still see that live today, but do it fast because it dies on the next code push, woot.

I have a sort of hit-list for what I think are DW’s five worst pieces of jank on mobile:

1. Quick-reply form trashes the layout on entry pages
2. New entry journal select overflows to damn Mars
3. Unnecessary Dramatic Woodchuck zoom in form fields
4. Control strip sloppiness, especially when logged out
5. Userpic browser compacts into neutron star (the “old-new” one in S2 pages, not the “new new” one on the new entry form)

This PR fixed #2 and one of the two worst instances of #3.

I have a PR in for #4 that should hopefully be uncontroversial. (The dangerous part of that actually got merged already bc it fixed an existing bug, ha. I think at this point it needs someone to test it on Browsers That Suck to make sure it’s not too bad a regression.) I also have another PR in for #1 (and the other bad bit of #3) — potentially more controversial, but it re-uses an existing design that people are hopefully comfortable with by now. #5 needs more research; there’s two avenues to fix it and the good one might be impractical for now, idk yet.

Beyond the hit-list, things get fuzzier. The site-scheme entry pages are probably the next worst, but they aren’t next in order because they get easier to deal with if some other stuff happens first. The default journal styles are mostly kind of crap on mobile in some very low-hanging fruit ways. I have some thoughts about comment nesting. We’ll see.