So it looks like during the past week or so whilst I’ve been away has been a relatively unstable one for the site. It boiled down to MySQL disappearing, resulting in Typo (Rails) dying.
I checked out `/var/mysqld.log` and there were a few entries with the following snippets:
051224 7:06:35 InnoDB: Fatal error: cannot allocate 8404992 bytes ofInnoDB: memory with malloc! Total allocated memoryInnoDB: by InnoDB 6319072 bytes. Operating system errno: 12InnoDB: Cannot continue operation!InnoDB: Check if you should increase the swap file orInnoDB: ulimits of your operating system.InnoDB: On FreeBSD check you have compiled the OS withInnoDB: a big enough maximum process size.InnoDB: Fatal error: cannot allocate the memory for the buffer pool051224 7:06:57 [ERROR] Can't init databases051224 7:07:00 [ERROR] Aborting051224 7:07:55 [Note] /usr/libexec/mysqld: Shutdown complete051224 07:08:02 mysqld ended
It looks like something is gobbling up memory to the point where MySQL can’t allocate enough. What’s weird is I can’t see what could possibly be trying to get MySQL to allocate 8.4MB? Any MySQL gurus know anything? Does MySQL just expand the data files by a certain margin every now and then, or is this allocation as a direct result of data manipulation/querying?
I was quite happily taking around 200+ hits per day a few weeks ago (although this seems to have dropped significantly as a result of my Google ranking for a few things dropping, related to outages perhaps?) so I’m guessing this is related to my (mis)configuration of Lighty and PHP, although I can’t quite see where.
I’ve changed my PHP FCGI socket configuration inside Lighty to recycle PHP processes sooner to see whether that helps:
"bin-environment" => ( "PHP_FCGI_CHILDREN" => "4", "PHP_FCGI_MAX_REQUESTS" => "100" )
Failing that, any suggestions from any Linux gurus? I’ve noticed that when it does die, it either completely locks-up the VPS, or, it seems to look absolutely fine without any process hogs. I’ve noticed that there’s been a couple of defunct PHP processes listed, is that indicative of a problem?
Anyway, hopefully this’ll sort itself following my tweaks, if not, any MySQL configuration changes people can suggest?
UPDATE: Seems like things seem to be running a little better now. I also compiled and deployed the latest release of Lightty to 1.4.11. So far it’s looking quite stable, no defunct processes to be found, and so far no hangs or crashes! However, following Robby Russell’s comments below, I think I’ll check out Postgres anyway!