Patchwork Psychology

I’m just finishing up a 65-hour week in which I was planning, executing and troubleshooting the first installation of a load balancer in my data center. To quell the rising traffic levels, it was decided to move one of the oldest customers of the company from a “highly-available” active/standby web/application server using the heartbeat daemon to a highly-available active/active web/application server using a load balancer. I had three hours of downtime to perform the cutover, which went smoothly, thanks to several hours of planning. I got home around 2am after a good long drive hanging out with Nat Sherman (or at least his delicious “Classic Mint” cigarettes). After tossing and turning for several hours, I was awakened by the call to come in because “the website’s down!”.

“The website’s running really slow,” one person told me as a congregation began around my cubicle. Honestly, the application is a hodge-podge of spaghetti code written by 12 different developers who never talked to each other, and as a result, it’s always slow. Every day someone calls about the website being slow. The only way to really fix that is to rewrite the whole thing. But they were apparently experiencing an unusually high call volume, so it was meritable.

So group of us started testing. After about an hour, we narrowed it down to something with SSL and non-text/html content types. When we tried opening a PDF (of which there were thousands), the file was transported at around 2K/s, no matter where in the network it was accessed (through the load balancer). When these files were accessed without SSL, they were transported at the maximum rate.

I then created an Apache RedirectCond and RedirectRule to redirect to the secure site for everything except .PDFs. After 20 quiet minutes, I decided to grab some lunch, which was interrupted by another call about the website being slow, to which I responded by removing the mandatory SSL policy.

In testing, I found some settings on the CP load balancer called “send buffer” and “receive buffer”, which were set to 16 and 4 (kb) respectively. I decided to try setting them both to 512. After the page came back I found them adjusted to the maximum of 256. I also found my problems gone!

Later I called up Coyote Point and had a chat with one of their engineers. After mentioning my problem the buffers were the first and only thing he had me adjust. I guess I should have called him first.

It’s noteworthy that he had me set the buffers to 32, not 256. He said the highest I should go is 128. The log shows some memory errors associated with having the buffer set at 256 until I tweaked it down to 128.

Well, in a few days I’ll try to put SSL back on and see for real if this is the solution.