View Full Version : [FQuest Alert] DEEDEE Drive Replacement
Terra
12-31-2003, 06:17 AM
At 4:37am tonight, DEEDEE kicked out its second drive from the RAID array...
We are already in motion to prep and replace the failed drive...
This will require a quick reboot as we perform the drive swap and bring the array back online...
Our sincerest apologies for any inconvenience this may cause...
--
Terra
sysAdmin
FutureQuest
Terra
12-31-2003, 07:31 AM
DEEDEE is now going offline to perform the swap...
--
Terra
sysAdmin
FutureQuest
Terra
12-31-2003, 08:03 AM
This is turning out to be more than a routine swap, as the 2nd drive is having issues...
The server is currently in 'single' run mode while we are working on lifting off the production data to a new set of drives...
More will be posted when we make further progress...
--
Terra
sysAdmin
FutureQuest
Terra
12-31-2003, 09:47 AM
We have just crossed the halfway point with copying client files out of the /big/dom tree... This is being done so that the files are as up to date as possible...
This is also being done because we are not 100% confident with the old drive and do not want it placed back into production...
So that left us with little choice but to start a laborious process of 'file-by-file' extraction from the remaining working drive of the failed RAID set...
Our sincerest apologies for the time required to perform this operation... I can assure you we are working as methodically and quickly as possible...
--
Terra
sysAdmin
FutureQuest
Wassercrats
12-31-2003, 10:35 AM
We have just crossed the halfway point with copying client files out of the /big/dom tree... This is being done so that the files are as up to date as possible...I remember reading stuff about FutureQuest making daily backups, but I didn't see it just now when I looked through the list of things FutureQuest offers. Does FutureQuest make daily backups? Any particular time of day/night?
Terra
12-31-2003, 10:45 AM
LOL - perhaps we should add that as a unique selling point... ;)
I mean doesn't every host make backups??? :P
We make nightly 'Buddy' backups, where one server is mirrored to the next...
This is all being transitioned soon though when we get the new Gigabit ethernet core network online, with a 4TB capable heavy duty monster taking the server flows... I cannot reveal too much about the project yet, however I can say the hardware is now inhouse and the project is waiting for a tad bit more stability out of the Linux 2.6 kernel...
--
Terra
sysAdmin
FutureQuest
ddunlap
12-31-2003, 11:07 AM
I absolutely hate to ask this question, but any eta on getting the files back online? I am getting phone calls. Thanks for continuing to work on it at, no doubt, breakneck speed, just need a time estimate to tell them.
Terra
12-31-2003, 11:18 AM
We are working on bringing it back online...
--
Terra
sysAdmin
FutureQuest
manfred
12-31-2003, 11:27 AM
Who is playing with the chair in the datacenter ?? :)
The datacenter looks like the bridge of the USS Enterprise (NCC - 1701)! The captain with the headset is looking at the WARP-Monitor :hehe:
Terra
12-31-2003, 11:42 AM
There was a last minute issue that forced us to pull DEEDEE back offline... We are working through it now...
--
Terra
sysAdmin
FutureQuest
Kevin
12-31-2003, 11:46 AM
Originally posted by manfred:
Who is playing with the chair in the datacenter ?? :)
The datacenter looks like the bridge of the USS Enterprise (NCC - 1701)! The captain with the headset is looking at the WARP-Monitor :hehe:
That would be me waiting for 20+ gigs of data to copy ;)
--
Kevin
Terra
12-31-2003, 11:48 AM
Who is playing with the chair in the datacenter
That would be Kevin, for whom is going well above the call of duty...
Thanks Kevin for being clutch player of the year!
--
Terra
sysAdmin
FutureQuest
Terra
12-31-2003, 12:19 PM
DEEDEE is now back online and in full production...
The extraction work only encountered 3 fatalities that were in non-critical sections of the data store... They were temporary files, so there was no loss in retrieved data...
Our sincerest apologies for the amount of time it took to extract the current data set, as I will always try this first before taking the easy way out of pulling 24 hour old stale data... I would rather take the hit with uptime credits to do this, then to explain to site owners that we have just lost your last 24 hours of file changes...
--
Terra
sysAdmin
FutureQuest
ddunlap
12-31-2003, 12:23 PM
All of you guys absolutely rock! And while I was waiting I made a nice little Excel file that shows what server all my accounts are on so I don't have to spend an hour trying to figure out who is affected :) . I have been meaning to do that for an entire year!
Happy New Year!
frankc
12-31-2003, 12:28 PM
Originally posted by ddunlap:
....I made a nice little Excel file that shows what server all my accounts are on so I don't have to spend an hour trying to figure out who is affectedJust a note to add that when I created folders for each website I manage in the Links, I rename it to add the server's name. Solves that problem nicely just by looking at the drop-down menu!
Chipmunk
12-31-2003, 01:11 PM
Originally posted by manfred:
The datacenter looks like the bridge of the USS Enterprise (NCC - 1701)! The captain with the headset is looking at the WARP-Monitor
Where would this picture be? You can't drop a :) Geek reference like that and not back it up (no pun intended) with a URL.
P.S. Just skimmed some of the old pics, and am wondering what
"Wolverine Anesthesia" (http://www.visionquest.net/misc/imagepages/image78.html) (rest obscured) is doing in a data center? Did the 'gov move their top secret R&D center from Alkali Lake to Florida, or is this something less sinister?!?
manfred
12-31-2003, 01:30 PM
It was the livecam http://www.visionquest.net/QuestVision.php?server=1&camera=2
Kevin did play with the chair in the middle :) Looked like he was sitting in a carousel (with a headset) and looking at the monitor. Maybe the monitor is the FluxCompensator of FQ´s datacenter!
By the way thank you Kevin for the fast and perfect work!
P.s.: Countdown for 2004 (in Austria): 5 hours 30 min.
Chipmunk
12-31-2003, 01:40 PM
Thanks Manfred - I don't suppose somebody has a copy in their cache, or perhaps FQ could dig it out? Come on Kevin, you :) want to share!
Bradley
12-31-2003, 03:54 PM
Originally posted by manfred:
It was the livecam http://www.visionquest.net/QuestVision.php?server=1&camera=2
Kevin did play with the chair in the middle :) Looked like he was sitting in a carousel (with a headset) and looking at the monitor. Maybe the monitor is the FluxCompensator of FQ´s datacenter!
By the way thank you Kevin for the fast and perfect work!
P.s.: Countdown for 2004 (in Austria): 5 hours 30 min.
Yeah I watch occasionally when he is working on my server to make sure there is no abuse going on ;) (it is a dell after all) and that chair looks like an amusement park ride sometimes lol
Kevin
12-31-2003, 04:04 PM
Originally posted by Chipmunk:
Thanks Manfred - I don't suppose somebody has a copy in their cache, or perhaps FQ could dig it out? Come on Kevin, you :) want to share!
Our camera systems are realtime only. They don't keep anything :(
--
Kevin
Kevin
12-31-2003, 04:05 PM
Originally posted by brnoe:
Yeah I watch occasionally when he is working on my server to make sure there is no abuse going on ;) (it is a dell after all) and that chair looks like an amusement park ride sometimes lol
Well I have to do something to amuse myself while data copies and computers boot. If I just sat and stared at the SCSI detection screen waiting to see if it sees the replacement drive I would end up pulling out all my hair like Terra :P
--
Kevin
Kevin
12-31-2003, 04:09 PM
Originally posted by manfred:
It was the livecam http://www.visionquest.net/QuestVision.php?server=1&camera=2
Kevin did play with the chair in the middle :) Looked like he was sitting in a carousel (with a headset) and looking at the monitor. Maybe the monitor is the FluxCompensator of FQ´s datacenter!
By the way thank you Kevin for the fast and perfect work!
P.s.: Countdown for 2004 (in Austria): 5 hours 30 min.
Actually the headset means that I was on the phone with Terra waiting patiently for him to do something on the server. The rest of the time I was wearing headphones connected to my iPod :)
--
Kevin
Bradley
12-31-2003, 04:12 PM
Originally posted by Kevin:
Actually the headset means that I was on the phone with Terra waiting patiently for him to do something on the server. The rest of the time I was wearing headphones connected to my iPod :)
--
Kevin
Gasp! Ipod?!?! How dare you! Actually I've got the 20gb model, which one you got? (mines the 2nd or 3rd generation model with the touch sensitive buttons.. kinda irritating at times)
and that chair looks like an amusement park ride sometimes lol LOOKS LIKE???? Only LOOKS LIKE????
Ha!
That's only because Kevin isn't "this high" yet for the REAL RIDES...which seem to take place within the home office on this end...
Folks... I hate to do this to you, but sometimes it seems you are bent on only 'assuming' reality rather than realizing that it IS Reality indeed...
Your CTO actually riding the chair powered by Max....
http://www.picolio.com/Albums/Max/MaxPullTerra.jpg
Deb
- Somebody has to define reality from fantasy...may as well be me :hehe:
Kevin
12-31-2003, 04:17 PM
Originally posted by brnoe:
Gasp! Ipod?!?! How dare you! Actually I've got the 20gb model, which one you got? (mines the 2nd or 3rd generation model with the touch sensitive buttons.. kinda irritating at times)
Mine is the 40GB model. I only got it about 2 months ago. Now I wish I had saved some money and gotten the 30GB model instead because if I put much more than 20GB of songs on it there is just too much stuff and I can't find anything. I tend to just do random play everything while I am on my bike so I only want good stuff on it. I have also used it as an external hard drive to transport data to other peoples homes.
--
Kevin
Kevin
12-31-2003, 04:18 PM
Originally posted by Deb:
LOOKS LIKE???? Only LOOKS LIKE????
Ha!
That's only because Kevin isn't "this high" yet for the REAL RIDES...which seem to take place within the home office on this end...
No, that's just because I can't take Bandit to the datacenter :(
--
Kevin
PaulKroll
12-31-2003, 05:50 PM
Your CTO actually riding the chair powered by Max....
Never mind that, look at all the computer books in that bookshelf... (Paul does his best Homer-drooling-sound).
Originally posted by PaulKroll:
Never mind that, look at all the computer books in that bookshelf... (Paul does his best Homer-drooling-sound). Selective Visuals to go along with that hearing eh Paul? I admire that 8}
LightGuide
12-31-2003, 07:40 PM
Originally posted by PaulKroll:
Never mind that, look at all the computer books in that bookshelf...
A man after my own heart... at last measure, 112 feet of shelf-space here is -- nothing but computer-related books, manuals, briefs, white papers, and journals.
Not that I've ever read a single one of them, mind you, but they certainly do look impressive.
etLux
I hope none of the titles end in "...for Dummies!"
Otherwise, we're all in trouble!:P
Happy New Year!
LightGuide
12-31-2003, 07:55 PM
Only the first ninety feet -- after that, I started to catch on.
etLux
Randall
12-31-2003, 08:06 PM
Do you have this one?
http://www.dummies.com/WileyCDA/DummiesTitle/productCd-0764553135.html
Randall
LightGuide
12-31-2003, 08:18 PM
No, I don't.
But I've ordered a copy of your all-time favorite for you...
this one (http://www.amazon.com/exec/obidos/tg/detail/-/1583940537/qid=1072916191/sr=1-15/ref=sr_1_15/102-2981085-0470568?v=glance&s=books)
etLux
Randall
12-31-2003, 08:37 PM
=)
But my all-time favorite is actually "Weiner Dog Art." Gotta love that one for the title alone.
Randall
Chipmunk
12-31-2003, 08:44 PM
Originally posted by Kevin:
No, that's just because I can't take Bandit to the datacenter
Hunh?!? You don't expect us to :) believe that FQ has a "no dogs in the datacenter" policy, do you? Or are they just picking on Bandit?
... I'm still waiting for somebody to answer my "Wolverine Anesthesia" (http://www.visionquest.net/misc/imagepages/image78.html) question.
Kevin
12-31-2003, 08:47 PM
Originally posted by Chipmunk:
Hunh?!? You don't expect us to :) believe that FQ has a "no dogs in the datacenter" policy, do you? Or are they just picking on Bandit?
... I'm still waiting for somebody to answer my "Wolverine Anesthesia" (http://www.visionquest.net/misc/imagepages/image78.html) question.
Actually I never asked because I assumed it would be a building rule since we are in an office building with other companies.
--
Kevin
Chipmunk
12-31-2003, 08:55 PM
Originally posted by Kevin:
Actually I never asked because I assumed it would be a building rule since we are in an office building with other companies.
Hmmm... you're in :) serious danger of losing Geek Points if you don't check the rules empirically.
Many yarns ago, I worked at a basement software startup by a couple who had two Irish Setters... who :) suddenly produced four puppies. All six lived in the basement :) with the programmers. Needless to say, liking dogs was listed on our employment ads, and was tested empirically during the onsite interview.
Hunh?!? You don't expect us to believe that FQ has a "no dogs in the datacenter" policy, do you? Or are they just picking on Bandit? We're just picking on anything with fur or feathers.... The day we put a cot in the DC for Kevin I'll include a reptile tank or something to keep him company ;) Needless to say, liking dogs was listed on our employment ads, and was tested empirically during the onsite interview. Excellent policy indeed!
Chipmunk
12-31-2003, 09:48 PM
Originally posted by Deb:
We're just picking on anything with fur or feathers.... The day we put a cot in the DC for Kevin I'll include a reptile tank or something to keep him company ;)
What, no :) fish tank? Ok, I personally think turtles are Nerdier than fish (no offense to Monty et al), so I suppose a reptile tank would be cool.
Excellent policy indeed!
That was more a :) fact of life.
I am considering adding a :) chipmunk compatibility test to my interview list, but that would :) limit hiring to a few months of the year. A few years ago, a friend (who I have on my mental list of potential sub-contractors) visited my former cabin with her writer husband, and while she & I were on one side of the cabin, he started shouting urgently. It turned out he had had the first chipmunk of his life sitting on his hand, stuffing a peanut (he had scoffed at me when I had forced a few on him "for your own protection"). Needless to say, I've been racking my brains since then for writing work for him.
P.S. Deb, I'm still waiting for an answer to my "Wolverine Anesthesia" (http://www.visionquest.net/misc/imagepages/image78.html) question.
I only know what Google tells me .. which is apparently more than Kevin knows :P
http://www.wacmdpa.com/
Now that I'm acknowledging what they must have access to though...... hmmmmm
Kevin, explain again why you spend so much time down there????
Kevin
12-31-2003, 11:02 PM
Originally posted by Deb:
I only know what Google tells me .. which is apparently more than Kevin knows :P
http://www.wacmdpa.com/
Now that I'm acknowledging what they must have access to though...... hmmmmm
Kevin, explain again why you spend so much time down there????
From what I hear they are a call center for an insurance company that covers anesthesiologists. They are also the only company in the building that has more than 10 people that are there every day.
--
Kevin
vBulletin® v3.6.8, Copyright ©2000-2012, Jelsoft Enterprises Ltd.