[FQuest Notice] MYSQL, MQS, and MMS server updates

[Kevin]

What..................: All MYSQL servers and all MQS servers that are currently running MySQL version 4.0.26 (most of them)
When..................: Tuesday, May 16th, 2006
Window...............: 12:00am - 3:00am ET
Estimated Duration: 10-15 minutes

All of the MYSQL servers and most of the MQS servers need to be upgraded to a newer version of MySQL. This is a security update which is why we are doing it on a short notice. The MQS servers will only be down for a few seconds and then only MySQL will be down. The MYSQL servers will be down for about 5 minutes each because they also require a reboot for a kernel upgrade to avoid a completely different security issue.

[Kevin]

This will also include MMS servers. They will require the kernel upgrade just like the MYSQL servers.

[Kevin]

This will begin in the next few minutes.

[Kevin]

All of the upgrades are done except for MYSQL01-05. Those 5 servers are having trouble and are not coming back up. It will be necessary to go into the Data Center to bring the servers back up. This may take an additional 30 minutes.

[chernove]

Thanks for the ongoing message updates.
Much appreciated, as always,
Eric

[Kevin]

OK, turns out the problem wasn't quite as bad as I thought it was.

MYSQL01-05 are now running again.

All upgrades for tonight are complete.

[Vairkjosa]

Just a quick ? If our servers were down tonight, cna we assume that they are done, or will they be down again?

[sheila]

Just a quick ? If our servers were down tonight, cna we assume that they are done, or will they be down again?
Yes. All done. :)

[Bradley]

Amazing, Kevin won the race of getting the update deployed first. As for me I think I see the finish line a few miles up :eeww: :P

Good job guys

[Paulo]

Hi,
What is the MySQL version that was used to make the update?

[Snarpy]

My CNC > Server Info says that MYSQL14 is running version 4.0.27.

[Paulo]

Gee, I should have looked there DUH.
Thank you so much.

[Kevin]

That is correct. All of the MYSQL##, MMS, and most of the MQS servers are now running 4.0.27.

The only MQS servers that aren't running 4.0.27 are the ones where different versions were specially requested.

[Grant]

Mine was down for an hour and a half!

(Or at least I received mysql error messages for that period!)

MQS0006

What happened?

Grant

[Kevin]

Mine was down for an hour and a half!

(Or at least I received mysql error messages for that period!)

MQS0006

What happened?

Grant
Your MySQL server wasn't actually down for that long. The upgrade on the MQS servers only took about 30 seconds however yours had an error in its startup script that caused it to break at about 01:22. I fixed that error and restarted it within a few minutes.

The error wasn't caused by the upgrade but it would have occoured whenever the database engine was restarted outside of a complete server reboot.

[Grant]

Interesting - cause I had 171 error messages, reasonably spaced out from 5:09 am to 6:27. So it looks a lot like it was out for that time - or a good chunk of it. It may have been up and down, but it was down a lot!

more or less the times:

5:09 AM, 5:09 5:13 20 20 20 22 22 22 22 23 25 26 26 257 27 27 28 29 31 31 31 32 33 33 33 33 34 35 35 35 35 37 38 43 47 48 48 48 52 55 56 56 57 57 57 57 57 57 6:01 01 02 04 08 08 10 13 13 14 14 15 21 21 21 21 21 21 21 21 21 21 21 21 21 22 then a whole lot of 22, then a lot of 23's, then 24 27 27 27 finally last 6:27

Lot of down time there.

Grant

[Kevin]

Grant,
The problem with your database was the the MYSQL engine was running with the CPU and memory limitations that are normally imposed on shell processes. That means that large queries would fail while smaller ones would work fine. That caused you to get some errors before we noticed the problem which took a bit since it wasn't really "down".

The surge of errors at the end was when I restarted the MySQL engine again after fixing the startup script. If it had actually been down all that time you would have seen that number of errors the entire time.

That startup script modification was something that I added to the MQS servers probably over a year ago to avoid this exact problem but for some reason it never made it onto your server.

[Grant]

Ok Kevin,

thanks for that - I wonder if there's anything else that didn't make it onto my server?

[Kevin]

I wonder if there's anything else that didn't make it onto my server?
I doubt it. I have been using this lately to work on all of the servers in parallel: http://sourceforge.net/projects/clusterssh

That is why they all went down at the exact same time instead of taking time for each one.