Roadrunner Twice

As mentioned previously, this year I switched to hosting eardogger.com in what's either a highly unconventional environment or an unusually conventional environment, depending on your perspective. This has mostly gone completely fine! However, I did have one incident several weeks ago, and it was a funny one.

I was out reading webcomics on my phone, and got creepy 500 errors on Eardogger; when I got home, the logs showed a Resource temporarily unavailable error when trying to access the database.

⁉️ (Metal Gear Solid guard alert noise)

All right, first off: That database isn't a remote server; it's a file on the local disk. If THAT's "unavailable," something's very wrong. A quick web search indicated that error comes from the operating system itself, not anything in my tech stack (like sqlite maybe). At some point, I visited a page on the site, then tried to run a command in my SSH session:

$ ls
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

Hahahaha holy shit.

Ok anyway, long story short: I was hitting my user's process limit and being prevented from spawning new processes or threads. The 500s were happening when concurrent DB reads would have made the reader pool spawn a new thread, and it hit the wall instead.

A given user may only run a certain number of processes at once on this server. Eardogger is at maximum a single process instance, so I thought I was fine. But then my web host upgraded my server's OS, which changed the process limit accounting to also include sub-process threads. And Eardogger IS multi-threaded.

How multi-threaded, exactly? Well, I was using the Tokio multi-threaded runtime with default configuration. And it turns out the default behavior is to immediately spawn one worker thread per logical CPU core...

...on what turns out to be a 128-core web server. The process limit (not advertised, but support will divulge if you ask) is 25.

I wouldn't want that even if it WAS allowed!! This app has like three goddamn users! I made my thread pool configurable and set it to single digits, and that immediately banished the errors and the shell lockups. 🌈 As a bonus, it also cut the app's cold startup time from "barely perceptible" to "legit gone" — apparently spawning more than a hundred threads on startup takes a noticeable amount of time, but Rust is so fast in general that it covered most of that sin and I wasn't immediately suspicious.

Lessons learned:

• Resource heuristics that inspect the local system's capacity are a huge red flag, because they act normal on your laptop and then go berserk elsewhere. Set your ceiling explicitly!
• top and ps on Linux don't list threads by default, you have to use an extra argument to see that.
• Wow.

EDIT: But actually, in most modern deployment scenarios, your server's CPU and memory resources really are much closer to the scene on your laptop, since the standard practice is to slice up computer resources into tiny single-purpose shards via containers or VMs. Like, Ruth was showing me something from her work that we would both consider "fairly extreme" in terms of resource allocation, and I was like "oh yeah, that's a whole lot of laptop... but it ain't a web server, you know?" So yes, this whole problem did in fact stem directly from my runtime environment being weirdly atavistic, and most people are probably fine with Tokio's default behavior.

Julia Evans’ StrangeLoop 2023 keynote was about digging into the different reasons a tool can be hard to learn, and it was a real good talk! At the end, as a tossed-off addendum to a conclusion about continuing to learn things, she said “I still don’t know why Git is hard.”

I happen to have thoughts about that one!

I have had to teach a fair number of (generally clever and persistent) people how to get around in Git or how to use its more advanced features. While doing so, I have often failed to get the basics to stick, which is incredibly aggravating to someone who prides themselves on explaining things. Sometimes this devolves into me talking about wave/particle duality as a crucial metaphor for getting through a rebase intact, and everyone in the room looking at me like my second head just tried to convert them to Gnosticism.

So, I’ve spent some time thinking about this before this weekend.

( Read more... )

Well I already posted about it on the one community where anyone's likely to give a damn, but I built a thing that'll stand up a working Dreamwidth dev instance on your laptop for you. Fill out the config file, vagrant up and answer "1" to its question, go into energy saver and tell your computer to not go to sleep for a while, and literally go out to lunch or something (installing perl modules is way slow for some damn reason), and you should have yourself a Fukken Livejournal when you get back.

The instructions on the wiki were honestly pretty dece, but there is no way in hell I was going to do that all manually. You never wanna be in a situation where if the shitbox VM goes down it takes part of your brain with it.

Why was I even doing this? Well, all the CSS offroading I did in my most recent journal style refresh taught me how to fix at least two or three of the top five (imo) most obnoxious pieces of Dreamwidth mobile jank. I've already got PRs in to fix two of them:

• Quick-reply form trashes the layout on mobile
• Massive trollface blowout on the create-entry page

...and across the two of them, I kind of snuck in a fix for another real stinker, where mobile Safari zooms the viewport to unusable hugeness as soon as you focus a form field. Maybe someone'll ask me to rebase that into a separate thing, idk.

Anyway, I feel powerful. 💪🏼

K, idk if y’all have ever messed around with the iOS “Shortcuts” app, but it turns out it is Legit.

So. On my Mac, I use a pair of oddball note-taking systems that I made out of baling wire, scrap lumber, and keyboard shortcuts over a decade ago and have been using daily ever since:

• “Garbage Book” is just a clone of Apple Notes,* except that it uses my normal text editor instead of a separate app that I don’t like as much.

• “FMP”** is, uh, baroque. There's a hotkey to get a text field that appends to a special text file. If I append a line that starts with something like ^didread, it’ll eventually get moved to a file like didread.txt (courtesy of a second hotkey that refreshes everything). And there’s yet another hotkey for quickly opening one of those note files by name.

  (I do realize what that all sounds like to a normal person, but trust me, it's a specialized ADHD workaround that solves some particular annoyances for me in a really sweet way.)

Since both sets of notes are just plain text files in a synced folder, I've always been able to access them on my phone. But it was way more awkward and slow, and I've always been on the lookout for ways to improve it.

Anyway, yesterday my dad was having brain surgery (he's doing fine, everything went great), and here's a thing about when someone close to you has major surgery: you sit around for a long-ass time, and you can't concentrate on any actual responsibilities for the whole duration (basically because the entire part of your mind that gives a flying fuck about responsibilities is wholly occupied with reacting whenever it's time to do something for said loved one). But a really frivolous and finicky project can be a great way to not obsess at 20,000 RPM over hypothetical stuff that you can't do anything about, so I decided to crack open that Shortcuts app that came out with iOS 12 and see if there was anything to it.

THERE IS. The card-based editing interface is weird as heck (it has some real strengths for programming on a touchscreen, but it also makes refactoring a pain in the ass, and was probably not intended for use on an iPhone SE), and the lack of some basic niceties like and/or expressions is pretty embarrassing (you can implement boolean logic yourself by adding more variables, but it looks like legit hell), but behind all that is a remarkably functional little scripting language with a coherent and consistent core and really unmatched discoverability.

It's probably even nicer for scripting apps that provide "real" Shortcuts actions, but even with just iA Writer's URL-based scripting and the built-in actions, I was able to completely*** re-implement both Garbage Book and FMP on my telephone over the course of two afternoons!

I now have little buttons on my "today" view for starting a new Garbage Book page, appending a line to my dump file, opening an FMP ^caret-tag file by name, and sorting ^caret-tag lines out of the dump file and into their real homes. (That was the one I didn't think would even be possible, but I totally did it!) And it all plays nicely with the edifice of Mac scripts that I've been using since '06.

Anyway, very satisfying results, a very impressive tool, also A+ distraction for that mandatory fret-in-the-hospital downtime.

_____

* Except this was ca. 2007 so it was more like a clone of Notational Velocity.

** Fast Memo Pencil? Fiendish Master Plan? Free Mashed Potatoes?

*** Wellll mostly completely. There's one thing I couldn't do on my phone from the original Garbage Book script: include part of the first line of text in the filename. That was always a nice feature, but c'est la vie; it's just a problem of different editing paradigms. In BBEdit I start by opening an empty editor window and typing for a while before hitting the Garbage Book hotkey to save, but in iA Writer I have to start by creating the file, so there's no first line of text to grab. I guess the editor itself could update the filename (Notes does this!), but it would have to know about my timestamped naming scheme to do that.

I finally switched from bash to zsh, and it was totally worth it.

I was paranoid about how much time it was going to take, but it turns out the trick is to do as little as possible, and most of your bash customizations will still work. It basically went like:

• Refactor your dotfiles to just isolate any bash-specific stuff that's in there. .bash_profile should consist of source ~/.bashrc; .bashrc should have your little handful of bash-specific configurations followed by source ~/.profile; and .profile should have all the aliases and simple functions that will work in both.
• Install oh-my-zsh and assemble a really simple .zshrc file. At the bottom, put source ~/.profile to catch all your old bash aliases and functions.
• Look through the list of themes on the wiki and find something that looks familiar enough that you won't feel weird. I went with muse.
• Maybe make one or two edits to that theme and save it as a custom one. I looked through lib/git.zsh and figured out how to add an indicator that my git branch is ahead or behind.

In the end, I got pretty much all I needed out of the box and only had to make about a four-line edit to the muse theme. Better tab completion, and I can spend about 90% less time running git status at work.

I realize this is of limited interest to this blog but oh well.

@Skud: not sure how i feel about http://app.net. i'm all for paid services to avoid values mismatch b/w users/advertisers/service operators
@Skud: but the sites i know that do that best (eg. @dreamwidth, @Pinboard) also have a very human/community feel that i don't see here
@nfagerlund: @Skud I feel that. W/ DW, I was immediately like “seems legit; I can tell who these folk are & how big it can get.” A.n is like Diaspora.
@nfagerlund: @Skud …by which I mean their story is critically incomplete in some way I can’t yet put my finger on.
@Skud: @nfagerlund yeah, i get that feeling too. which is not necessarily an impediment to their platform taking off...
@Skud: @nfagerlund ... but does mean that my gut feeling about them is a bit nervous

Yeah, so skud was doubting on app.net, and I am doubting on it too, and I don't have much more to say about said putative service at the moment. But I'm posting because I think I finally DID put my finger on what was wrong with Diaspora! And I am very proud of myself for it.

So Diaspora was meant to be Facebook without all the evil, right? Here's the problem with that: Facebook without the evil is NOTHING.

Because what the hell even IS Facebook? The answer changes significantly every nine or ten months. I joined it in January 2005 because it was a visual address book to my college, and I needed that when I was looking for a new sublet. Then it turned into a walled-garden email replacement. Then it turned into Flickr for spring break photos, then it finally managed to replace the .plan file, then it was also Livejournal for about a month, then it tried to be Craigslist for a summer, then it was Twitter with less focus, then it was a platform for shitty little games that you pay real money in order to not have to play. Now it's just sort of an undifferentiated mishmash, although it seems to be turning into Tumblr lately, mostly to accomodate George Takei.

Obviously there is no common thread. Facebook's soul is not in what it does for you or allows you to do. The product itself, the THING that Diaspora tried to copy, is frankly irrelevant. The one thing that has always made Facebook Facebook is that fucking practically everybody you know is on the goddamn thing, and they got there because Facebook was persistently and craftily evil.

The original short-lived college-scope restrictions on the thing were brilliant, because they made people let their guards down, join up, and put their personal info in. That made it easier for peoples' friends in other colleges to find them, which anchored them further in. All the effort to make it difficult to add people to your normal email address book meant that you were signing in on a regular basis to get/send messages, and would be more likely to see new friend requests and other activity, which would keep you interested. Let's not even get into that Zynga Skinner box. Etc. etc. etc. The only reason you're on Facebook now is because they were evil, and although you'd leave if your friends left, they're all still on there because of the evil. The fact that everyone is there makes Facebook a horrible place most of the time, but it also makes it indispensable, and the fact that you can't properly export your information makes it non-disposable and non-replaceable.

I don't think the people behind Diaspora ever understood any of that. They thought people were on Facebook because Facebook was a good app, and people actually wanted some atrocity that was kind of like Tumblr/Flickr/Twitter/LJ/toilet-graffiti/emotionally-abusive-Gameboy except worse. That's manifestly not the case. People want everyone they know in one place, and the only way to give them that is to be evil. Which makes it impossible to replace Facebook with any less-evil alternative -- whatever eventually kills Facebook will win by being either MORE evil, or more SOPHISTICATEDLY evil. And since Diaspora was unable to compete with Facebook, it found itself competing with all the non-Facebook focussed-purpose services like Twitter and Flickr and DW and Tumblr, and it since it was built to be worse than all of them, you probably still aren't using it. Of course, you're probably not using Dreamwidth, either. You probably ARE using Twitter, and I'll be interested to hear app.net's plan for dealing with the fact that 80% of Twitter joined Twitter because all their friends were on Twitter.

Yes, this is depressing and annoying. Whatever, let me have my moment of explanatory triumph.

I like using spoiler text! Spoiler text doesn't like iOS. Or vice-versa, whatever.

Anyway, I found a trick today that works great on both desktop and mobile, and which will work without Javascript, which means I can use it on my Dreamwidth! Unfortunately, you don't get the benefit unless you're viewing the entry in my style, and I don't get the benefit unless you write your spoiler tags such that they'll work the same way on my reading page. Frustrating. But it degrades gracefully to normal-acting spoiler text, so what the hell, I'll use it anyway.

It goes a little like this:

Go here to add custom CSS to your style, and add the following:

a.spoiler, a.spoiler:link, a.spoiler:visited {color: #3a3a3a; background-color: #3a3a3a; text-decoration: none;}
a.spoiler:active, a.spoiler:hover {color: #000 !important; background-color: #e7e7e7 !important; text-decoration: none;}

Then, you can make spoiler text like this:

Normal text, <a href="##" class="spoiler" style="color: #3a3a3a; background-color: #3a3a3a">spoiler text</a>, normal text.

Normal text, spoiler text, normal text.

Honestly they should have put a <spoiler> tag in HTML 5, but this will do for now. Sorta. I guess. And no, I don't know how to make spoiler text work with both screenreaders and iPads. :(

So I grabbed a new iPad text editor to play with! It’s called iaWriter, and you might have heard of it, since apparently it’s the one Stephen Fry uses or something. I like it a lot. It can’t replace PlainText, because its syncing and organizing are kind of barely sufficient (verging on crappy); it’s completely unusable for making frequent updates to a blizzard of smallish documents, and appears to be meant for working diligently on four or five large files, which will occasionally be switched out. It’s a text drafter, I think, not a text editor. Anyway, the upshot is that I seem to be able to write fiction with it at speed, which is one hell of a feature and which I always found too frustrating to keep trying for on Simplenote and PlainText.

The two real standout touches are that it uses a big ugly font that’s very readable no matter how I’m holding the device, and it extends the iPad keyboard with a row of extra keys – things like always-on apostrophe and quote keys, and cursor keys that can navigate by letter or word. It sounds trivial, but it ends up making a pretty huge difference, and I’m inclined to attribute its surprising usefulness to the amount of friction that takes away. I realize most iPad apps can’t do that with the keyboard, because it makes it take up like 3/5 of the whole screen, but it’s damn nice if you need to type for an hour.

It also has two standout annoyances: there’s a big old scrolling dead zone on either edge of the page, and it doesn’t switch the keys back after you type an apostrophe with the normal punctuation keyboard. Yes, you have a built-in key for that, but that’s no reason to punish me for picking up a little muscle memory elsewhere, geez.

I think I paid like a buck for it? It’s definitely worth checking out, even if you already have a main text editor.

(Oh, and apropos of nothing, apparently it saves newly created files as Little-Endian UTF-16. Who in the world uses that for ANYTHING? Weird!)

Oh hey, what the fuck, Tweetie 2 I mean Twitter for Mac finally came out? That was unexpected. Well, if there's one thing I love, it's being kind of a bitch about Twitter clients, so let's ride:

• It's got support for lists! I think it's the first Twitter app for Mac that does the list thing and yet doesn't seem misguided and broken on a totally fundamental level. (Nambu's probably the closest behind, but I really can't stand it.)
  • Unfortunately, it doesn't expose them flatly the way Twitterrific for iPad does, so you have to drill down every time to get to them. Bogus.
• Not down with this slide-and-stack UI nonsense they've got going on. Honestly, what the hell is supposed to be happening in this metaphor? That this stack keeps getting taller and taller every time I change views makes me nervous, like I'm going to have to clean it all up someday when the pile tips over and gets all over the foyer. I am only 30% joking, here.
  • On the other hand, I thought the Twitter for iPad UI was godawful at first, but they improved it pretty dramatically in the minor updates. Maybe they'll make it more spatially coherent in a month or three.
• I've gotten used to Itsy's inline images, which make TfM's popups seem rather outdated in comparison.
• Ha—when I unplugged my monitor, the window stayed taller than my MacBook's screen; now it won't auto-shrink, and there's no way to grab the resize control. Enjoy the land beyond the bottom of the screen, little window! Go where no man has gone before!
• Hmm, the keyboard shortcuts seem pretty rich. That's nice.
• It's still got the ability to do multiple tweet windows, which was always nice to have when I needed it. Does any other app else do that, or is that still Tweetie-only?
• But DO NOT WANT tweet windows that hover over the top of everything. Eew. I understand the problem they were trying to solve, but I disapprove. I'd prefer that they come to the fore whenever any Twitter window was active but act like normal windows the rest of the time.
• It's still tough to tell whether the main window is active, but that's less annoying than it was when it looked like it should dim the way normal Leopard/Snow Leopard windows do.
• Feels pretty fast!
• I can't remember: Was there ever any actual apology for that debacle where MacHeist and Atebits sold vaporware sneak peeks of Tweetie 2 and basically never delivered? *checks* Hahaha—looks like there's this, and also that, for whatever those're worth.

Anyway, all told I'd say it's a pretty decent app. I'm using it for now. Still hoping Twitterrific for Mac ends up living up to the promise of the iPad version. (Man, come to think of it, that update's been pending for even longer than Tweetie 2, hasn't it?)

I learned how to talk to Growl today! Hello, Growl.

Context here is that I'm trying to abandon ship on Quicksilver on account of its oldness, staleness, and general decrepitude, but since I used it for such a weird variety of stuff, I'm having to get a little creative on how to replace it. In this case, I'd rigged QS to throw a Growl alert describing the currently playing track in iTunes whenever I whacked a certain hotkey, and since I wasn't interested in running a whole extra app just for that, I figured I could probably roll my own thingamabob and trigger it with FastScripts.

I figured right! ( And here's the goods: )

For reasons that are somewhat complicated, you'll have to paste that into AppleScript Editor and save it, then uncomment that one line, run the script a single time, and close the file without saving. After that, it'll work just fine, and you can set the display options for the alert in Growl's preference pane; the name of the application will be "iTunes Track Alert."

(Yes, I could have just left that line uncommented and skipped that extra step, but that seemed to make it run slower when I was testing it, and it really does only have to happen once.)

Incidentally, my hacked-up-in-an-afternoon-bullshit-AppleScript replacement for this feature turns out to be better than Quicksilver's version, because it shows the artwork even when it's not embedded in the file and can display the track's rating. Take that!