[FQuest Alert] SIX TimeWarp

[Terra]

Earlier this morning, there was a backwards time jump with the SIX server's time clock...
12/Aug/2003:07:49:57 ==> 22/Jul/2003:14:22:21

The system time was corrected at: 12/Aug/2003:08:07:39

We are not sure how or why the jump occurred, but we are investigating both the time synchronization software as well as the hardware (RTC) clock...

We have also removed the erroneous entries from any Apache log files, so as to not perturb the STATS processing...

If you are using scripts that timestamp important data, please look for any erroneous entries in the time range:
22/Jul/2003:14:22:21 - 22/Jul/2003:14:36:31

that would have gone to a logfile intermixed with log entries from: 08/12/2003

Our sincerest apologies for the confusion this event may have caused...

--
Terra
sysAdmin
FutureQuest

[Beckles]

I'm sure I'll sound like a dolt to some for not knowing what servers my domains are on, but if you know we have an account on SIX (which I assume you do since that's what the email I received said), I really wish you'd tell us which account. I know I can see what server an account is on by going to the CNC, but I have four different domains here at FQ and I would think it would be a lot easier for you to just tell me which account(s) was affected than me having to check all four .... Just a suggestion for future such incidents. :)

[Terra]

The message was designed to be generic, as we did not want to add the overhead of trying to add specifics on a per recipient basis...

In short, it was meant to be: K.I.S.S.

--
Terra
--would not want to be the black pot sitting next to the kettle--
FutureQuest

[frankc]

Originally posted by Beckles:
I'm sure I'll sound like a dolt to some for not knowing what servers my domains are on....Nah, it's just one more thing to remember. To solve that, in my favorites/bookmark list, I modified the folder name to add the server name AND the IP address for each of the domains. Very handy.

[Bob]

FWIW...

I keep an Excel spread sheet that has each full package listed along with the username, Server and IP address. I also list the Package password and all IRs under that package.

I update it everytime I make a change and print it out for a paper backup as well as the saved file and have it right at hand on my desk for easy and quick reference.

-Bob

- Never hurts to have information in multiple locations :yeah: -

[Beckles]

Terra,

I understand why you did it, but my point was you may want to think about a way of providing more useful notifications. As a customer of your company with four accounts who pays a good bit of money each month and year for those services, I was a bit disappointed you sent me an email basically saying "one or more of your accounts had a problem, if you want to know which one log into CNC for each account and find out yourself." As your customer I should hope that your primary goal in operating FQ is not to keep everything simple for your convenience. I was just trying to provide a suggestion for future such situations, yet your answer was "we just did what was easiest for us." I hope you can understand that as a customer I find that answer a bit disappointing. You would have been much better off saying nothing ... all I was really hoping for was "We'll think about a better way of doing such notifications in the future." ... no need to promise you'll actually do anything, I know you guys are busy.

It's not a big deal, but it is something that matters to me, obviously, since I took the time to post it here, and I was simply trying to provide a suggestion that you might want to consider for future situations. It would seem to me that if you pulled a list of everyone on a specific server you should be able to also pull their domains on that server. Another option would be to send an email to webmaster@domain.com for each domain on an affected server, then we could see what domain was affected just by looking at the email address it was sent to.

[Deb]

I do agree that including the affected domains would be helpful in the emails... though we would not send to webmaster@domain.com (think reseller's and the like who would rather not have their clients receiving such notices from us).

I also agree that keeping track of these things early on can be helpful as there are times when a 'list of domains' wont be sent along for a variety of reasons.

In short, it is something we can work toward, and in the meantime, it would be good for clients to use whichever local method works best for them for those times when specific domains are not listed. As an example a general forum post concerning a server upgrade that doesn't include an email... or when all servers are being upgraded over a period of time and you want to know which accounts will be affected on which dates.

Deb
- Tagged and Noted

[Beckles]

in the meantime, it would be good for clients to use whichever local method works best for them for those times when specific domains are not listed. As an example a general forum post concerning a server upgrade that doesn't include an email... or when all servers are being upgraded over a period of time and you want to know which accounts will be affected on which dates.

Deb, note that I have no problem on more "general" items not including specific domain information ... those are much less time sensitive. In this case though, when you take the time to send me an email about a specific problem (which is very rare in my four plus years of being an FQ customer), it must be pretty ciritcal and/or unique problem so the fasterI can get the information the better. Right now I'm at work and I honestly don't have access to all my information on my domains ... I just don't bring it with me to my work.

[Terra]

"we just did what was easiest for us."
That is not what I was trying to convey..

1) To get a notice out ASAP to all affected site owners on the SIX server, since this was a 'timeliness' issue...
2) We highly frown on 'per recipient' customized email lists, therefore my tagline (pot call kettle black) was a point mentioned that we will always try to abide by the same rules and stipulations that our clients should work under as well... It would appear hypocritical if we do our mailing lists otherwise...

I will agree that my post was a bit terse, as I was consumed in fighting another fire at the time... Though I did want to get a response out to you that was direct and to the point... Sometimes my social skills fall a bit from grace when battling a 7 alarm fire...

--
Terra
--it won't be the first time Deb has threatened to take my keyboard away from me--
FutureQuest

[Deb]

Right now I'm at work and I honestly don't have access to all my information on my domains ... I just don't bring it with me to my work. I've sent along an additional email to you which specifies which domains are on which servers.

Deb
- Assist...Assist!

[Beckles]

Originally posted by Terra:
"we just did what was easiest for us."
That is not what I was trying to convey..

1) To get a notice out ASAP to all affected site owners on the SIX server, since this was a 'timeliness' issue...
2) We highly frown on 'per recipient' customized email lists, therefore my tagline (pot call kettle black) was a point mentioned that we will always try to abide by the same rules and stipulations that our clients should work under as well... It would appear hypocritical if we do our mailing lists otherwise...


Terra, I totally understand ... and I'm really not trying to be a jerk about this, but I feel you're underestimating the importance of what I'm saying which is the only reason I'm responding at this point just to provide my side:

1) I agree it's a timliness issue ... which is all the more reason I need that additional information in the email and not have to get it myself.

2) If you're so concerned about sending out customized emails, you really need to examine your overall business practices. I honestly did not realize it was such an issue. I receive two personalized invoices per month and two personalized receipts per month. Seeing that I have automatic credit card billing, I really have no need to receive both personalized invoices and receipts. I would suggest you eliminate the 2 personalized invoices you send me each month to clear overhead to be able to send out personalized server notices on the rare instances where you make them. You say you "highly frown" on the practice, implying that you leave room for such cases where it is necessary to personalize a larger mailing .... like I pointed out you do it every month for invoices and receipts already. If it's important and timely enough for you to send me an email about it (instead of just posting it on the forums), I think it's probably important and timely enough for you to personalize it.

[Terra]

I do not underestimate your request, which is why I tried to convey the 'real' reason why we elected to not do it that way...

The billing system runs invoices and receipts at special times of the evening when traffic is low... Or we may send an updated and personalized invoice/receipt on a 'one-off' manual send... The billing system also has logic to not send if there is even a hint of any backlog in the queues... The first priority is to get the clients email through, with our email taking lower priority...

There is a lot of 'behind-the-scenes' technical resource allocations and methods that are not realized by site owners... I perhaps should have conveyed that in my first post to avoid any potential miscommunication...

Your 'idea' will be placed on our internal discussion list to see if we can do an 'Emergency/Alert' email that is personalized for each site owner... We have approached this in the past, because I myself want what you have suggested, however for various technical, time, and resource issues we have decided against it...

And finally, this is such a rare event - sending an email focused only to one server's site owners - that I did not even foresee the eventual backlash from getting the alert out quick...

--
Terra
--will just need to make sure we never have another (specific) server problem - but it will still be OK to break them all at once-- :P
FutureQuest

[YFS200]

Never had a clock do that before. Least when the computer is still on.
Looks like SIX has still not recovered.
Current Server Status
15 Minute Load Average:7.00

Keep up the good work.

[Terra]

SIX is still having intermittent issues, and has been the focus all morning...

Most likely I will pull its plug and swap in a new server...

--
Terra
--No mercy--
FutureQuest

[YFS200]

11:50am up 19:43, 7 users, load average: 22.79, 19.71, 12.68

It's SIX stress test day!!!

[kitchin]

If you want to have a page with system info on it, outside of CNC, upload this file, called say, "info.php", to your www directory:

<?php phpinfo(); ?>

Then you can just go to http://yourdomain.tld/info.php when you need it. Whether you password protect it is up to you...

The name of the machine is in the first entry:
Server: Linux astro.futurequest.net ...

[Monty]

think reseller's and the like who would rather not have their clients receiving such notices from us
I absoutely don't want any emails going out to my resold accounts to such an address and like Bob, keep a spreadsheet with all my accounts on a floppy. It took me a matter of seconds to track down the affected domains and even the domains on SIX weren't affected as they were static .html pages. Please don't change a thing, it worked perfectly for me this morning.