Earlier this week, I and my 3.8 billion scraper-bot friends noticed that all my websites were down. I’d been meaning for months cough to migrate my elderly and infirm server to something that didn’t look like an 8th grade Biology frog. (There’s a photo I should recover and post, but it’s not happening right now.)
I’ve lost track of how many restores I’ve done for the boot volume, so it’s well past due. This time it was web data. I’ve admittedly been lax in fixing my MySQL backup automation, so the backup I had was laughably old. Yet Time Machine had never failed me before, so I figured if it came to that I’d sort it out somehow. Kinda. More-or-less.
You can see where this is going.
Now, in fairness to Time Machine, I’m not sure if it was the problem. It seems to have been perfectly fine for everything else, and databases are not exactly friendly to backup services. I had directories of healthy-looking frm, MYI, and MYD files, which in theory can be pasted back together.
But it looked as if they were last updated nearly a year ago. (Not sure where those months of posts were stored in the meantime.) For most of my blogs, that isn’t a big deal. This one, however, is a different story. But I’m getting a little ahead of myself.
I first tried to restore via Time Machine to a fresh drive, which involved finding a USB hub so I could have it and the backup volume attached at the same time. I also formatted the new volume on the same machine, to avoid any problems with using a more recent one. (To put this in perspective: the host is OS X Server 10.5, on hardware of its era, with the web volume mounted by FireWire 400.)
It took many hours of semi-skilled poking around to confirm that I had what looked like could be made into a usable database. All the other files restored fine, but MySQL was in a bad way, and took WordPress down with it. I could drop my directory in the default location on the boot volume (/var/mysql), manually start MySQL, and see the data in the mysql interactive shell.
But if I tried to change the configuration to use the real location, I only got errors about a pid file not being updated and then the process quit. And in any case, WordPress had no idea where to find this strange new MySQL.
Everything had been working with ServerAdmin and launchd, the LaunchDaemons plist configured with appropriate options and starting /usr/libexec/mysqld (not something in /usr/bin, but as an OS X Server person I’m used to this.) Except the process wouldn’t start, complaining it couldn’t find a msql-bin.index that was right there in front of it. I even checked with dtruss (OS X strace) and I could see it changing to the directory and then failing to find the file that now had world-everything permissions.
Then I tried a database dump to see if I could import what I had into a test WordPress install. It wasn’t pretty (the original was running 3.8) but it did semi-function. That was promising.
I set up a hosted VM (like I had been planning to all along) to use as a new webserver. Installed a whole ton of things, did some DNS foolery, and then copied over both the db dump and my blog directory. I couldn’t import the db directly into a more recent WordPress version, so I tried to drop the existing 3.8 into place. That failed because the new server (Ubuntu Xenial) didn’t have necessary php5 (!) things the old WordPress expected.
So.
The next option was to import the dump into a new database with similar settings, make a fresh copy of the old WordPress dir, and immediately upgrade a recent version on top of it. This more-or-less worked. I had to manually update urls in the database, now that it was on a different host, and fix mod_rewrite paths in .htaccess.
Just to note here: if you ever find yourself needing to do unlikely things inside the guts of your WordPress database, please tread carefully with the sql commands. It is very easy to screw up, well, basically everything related to your db. Including MySQL itself.
At last I had a functioning blog, although one that was woefully out of date. It may be that there’s a way to recover that other data, but in the meantime I have to live with the db I have. I went searching the Wayback Machine and found nearly everything. Thank you, Brewster Kahle.
ProTip: If you find yourself recovering blog posts from The Internet Archive, view the page source so you can copy the HTML-ified text that came straight from scraping your site. I had a lot of creative formatting in those recent posts, so it helped a lot.
At this point, I have one blog recovered on the new machine, although not with the proper hostname. So it’s time to set up Apache virtual hosts again. I did that, and then switched my feorlen.org DNS to point to the new server. (This involved some more db fiddling, but whatever. One day I’ll learn to update WordPress settings before yanking things out from under it and not after.)
It was all working, until then it wasn’t.
Right now, as I’m typing this, suddenly the hostname that was resolving to the new server is hitting the old one. And I have no idea why. It’s probably some weird DNS propagation thing, so I should not get worried about it for a few more hours. I’m saving this text locally in case it all blows up.
At the end of all of this (going on three days) I’ve got one blog recovered, two domains on the new server, and one other blog that appears to be recoverable by the same process. (Several others are long dead, so aren’t getting moved.) Oh, and in the middle my everyday laptop started acting weird, and so did the refrigerator. Just because.
I’ve lost one non-public page, really annoying because it’s the progress report for a project I did, and a couple blog comments. And it was very much a giant pain in the ass. As soon as I get all this settled, you can be sure I’ll be working on that db backup automation.
The things I was reminded of are:
First, this success story was brought to you by a pile of random parts. The half-dead retired USB hub. FireWire 400 cables. Old machines I could safely test restore Time Machine backups to. If you run old hardware, you have to have similar vintage accessories available to be able to work on it when things go badly. And they will go badly. Newer machines may or may not boot that old volume, and the old one is very likely to choke on something too recent. (Time Machine has cheerfully put up with the many times I’ve restored across multiple major OS versions and other sorts of nonsense.)
Second, The Internet Archive is amazing and you should give them money. I should too (when next budget opportunity comes around.) I wrote a bunch of long blog posts in the past year, and they saved my bacon.
Third: Do Your Damn Backups!! And not half-assed when-you-get-around-to-it, either. Pretty clicky-clicky methods are fine for desktops, but servers are a different matter. Databases are another thing altogether. I know all of this, had several good general backups, and still was lazy and didn’t do enough database backups. Don’t be me.
Leave a Reply