I've got lots more to write about what's been going on in my life lately, but it's 1AM, so I need to finish updating my Mac Mini server and get to bed.
Update: Hahahaahaha. I am still sitting here at 3AM with my eyes glazed over, smiling like a half-wit at the last sentence I wrote: "it's 1AM, so I need to finish updating my Mac Mini server and get to bed." Famous last words, indeed.
It turns out that one of the Apple updates I downloaded overwrote my specially-compiled php libraries and interpreter. It took me the past two hours to come to that conclusion. :( The stock version of php that Apple gave me this evening was obviously not compiled for fastcgi (which is to be expected). This lack of fastcgi support, in turn, was preventing lighttpd's mod_fastcgi from ever creating a php fastcgi process.
The hard part in debugging this debacle was where to start, as I had just finished uploading tons of code changes and bugfixes. First I suspected a rails routing loop because I was seeing my .php requests being logged over and over again. Eventually I had enough common sense to enable fastcgi.debug, which then showed me that the php fastcgi process was failing to launch correctly. This explained why I was seeing the requests being logged repeatedly: lighttpd was trying and re-trying the request furiously while the php process continued to bonk. At that point, my sleep-deprived brain grudgingly made the connection between that darned Apple update I ran just before going to bed. At least, I thought I was running the update "just before" going to bed.
Lesson learned: do not run your operating system's automatic update process after a major application upgrade at 1AM in the morning.
The Christian right continues to amaze me with their vile potion of ethnocentrism, fear, and reactionary hatred. As a Christian, I just want to clarify for anyone who may be wondering: this sort of fear/war-mongering is about as far from the teachings of Christ as you could get. I suppose I shouldn't be so surprised. I was talking to my brother-in-law (who is a pastor in Minnesota) about this, and as soon as I mentioned the name Robertson, he replied, "How did he embarass Jesus this time?"
[Sun Aug 14 20:07:38 2005] [error] [client 192.168.0.15] Provider encountered an error while
streaming a REPORT response. [500, #0]
[Sun Aug 14 20:07:38 2005] [error] [client 192.168.0.15] A failure occurred while driving the
update report editor [500, #160010]
[Sun Aug 14 20:19:04 2005] [error] [client 192.168.0.15] No such string '7o' [500, #160010]
Obviously, I had no clue why this was happening--I didn't remember anything odd going on with my subversion server in the past few weeks. Then again, it has been so long since my last check-in that I'm not sure I could have remembered. So I searched a little bit for this business about the report editor and "no such string XX". I found a few newsgroup postings, but they all seemed to be dealing with specific cases. The subversion FAQ caused me some momentary panic when I read about how the "failure occurred while driving the update report editor" error might require me to rebuild apache and mod_svn from source. Some more reading and testing revealed that I was (thankfully) not suffering from this problem.
In the end, good old 'svnadmin recover' saved the day:
Doug-Fales-Computer:~doug$ svnadmin recover /Library/projects/ Repository lock acquired. Please wait; recovering the repository may take some time... Recovery completed. The latest repos revision is 16.I had to adjust some permissions (I had run svnadmin as a different user than the one that httpd uses to access my repository), but then everything ran great again.
I'm not sure if I should feel good about 'svnadmin recover' fixing the problem, or if I should be disappointed in a one-user, one-project subversion repository becoming corrupt for no apparent reason. :/