[FQuest Notice] FutureQuest Secure_Mode™ Deployment - TakeII

[Terra]

When: 5/23/2003
Window: 1:00am - 6:00am (EDT)
Duration: 5 - 60 minutes (per server)
Servers: ENIGMA, RASMUS, ASTRO

Please view the original announcement at:
http://www.aota.net/forums/showthread.php?s=&threadid=14277

The issue that caused the postponement was tracked down to a rare condition between Apache and the Linux SysV Semaphore capabilities... It was completely unrelated to our Secure_Mode™ development and was just an unfortunate coincidence that backed us into a corner where we had to play it safe...

The problem with the Apache mutexes has been fixed and the new engines have been in operation for several days now without incident...

--
Terra
-- http://www.aota.net/4F/ThrillIssues.mp3 --
FutureQuest

[rpiwireless]

That's great! Thank's.

[phppete]

Hi Terra,

The values:

FQ_Secure_Mode_group apache
FQ_Secure_Mode_user apache

When FQ_Secure_Mode is on can we change these values with .htaccess?
What user and group will folder and files run under when I create them with PHP?

Thanks

Pete

[Terra]

1) Those are locked read-only values... It would be a serious security hole for it to be any other way...
2) Your uid and gid, and not 'apache:apache'...

--
Terra
--Who did you want to be Today?--
FutureQuest

[mromero]

So I am free to go in and try to setup Gallery on my own now?

Regards

[Terra]

So I am free to go in and try to setup Gallery on my own now?
:umm:

The upgrade is not scheduled until early Friday morning... Also, the domain you had not referred to must be on one of the scheduled servers being upgraded...

--
Terra
--read it 10 different ways and is still confused--
FutureQuest

[Terra]

ENIGMA's Apache daemon(s) will be taken offline in a few minutes to begin the conversion...

--
Terra
sysAdmin
FutureQuest

[Terra]

ENIGMA is back in full production with all conversion work done...

So far, everything appears to be running smoothly...

I will give it a few minutes to settle in before proceeding forward with RASMUS...

--
Terra
sysAdmin
FutureQuest

[Terra]

RASMUS will now undergo the conversion procedure...

--
Terra
sysAdmin
FutureQuest

[Terra]

The RASMUS conversion is now complete and all Apaches daemons are back in production...

Moving onward to ASTRO...

--
Terra
sysAdmin
FutureQuest

[Terra]

The ASTRO server conversion work is now completed...

All scheduled servers are now upgraded and back in full production...

Please let us know if you encounter any difficulties with your PHP scripts...

Enjoy! :)

--
Terra
sysAdmin
FutureQuest

[one2net]

Congratulation for this great and hard work!!
Any scheduled date for the QBERT conversion ?

[Terra]

Right now I'm playing it by ear... If there are no problems today with what is already deployed, I will line up some more (TBD) servers...

--
Terra
sysAdmin
FutureQuest

[one2net]

Looking foward to it.
Thanks a lot.

PG

[Crow]

Any idea when Hanna is gonna be done??:\

[Terra]

The Secure_Mode™ deployment is currently suspended due to the following two issues:
1) http://www.aota.net/forums/showthread.php?s=&threadid=14423
2) http://www.aota.net/forums/showthread.php?postid=88957#post88957

#1 appears to have cleared up, as no further 'D state' stuck Apaches have been seen... Looks like it was isolated to a bad interaction between PHP sessions and tmpfs - however I want at least 2 to 4 weeks runtime to ensure the problem in fact has gone away for good... What I can tell you is that this condition is severe enough, that I'm not going to take any chances with it... When this bug rears it's ugly head, it causes a debilitating domino effect to the point of being forced to reboot the server...

#2 we are working night and day to reengineer certain sections of Apache and PHP to make sure that when PHP is switching it's userid on the fly that it won't cause Apache to put forth a spurious and random 403 (access denied) error... What we do have in place are traps to catch when it is happening, however the interactions leading up to it are so differing and complex - that isolating the common conditions has been very difficult...

With both of the above errors, none of this showed up in Beta testing, that ran for close to 6 months... The torrent of the Internet, once Secure_Mode™ was put into production, has caused a few rare corner cases to surface...

These issues must be addressed before we can go any further... Thus far, they are the only 2 outstanding problems that we are confronted with... Everything else is working rather well, with millions and millions of successful PHP requests served...

And that is the current Secure_Mode™ state of the union right now...

--
Terra
--stability is job #1--
FutureQuest

[Crow]

Is there any new news?
(Hanna Server):@

[Terra]

Is there any new news?
Work is already underway to merge and test our changes with the PHP 4.3.2 tree...

The existing problems that halted the deployment last time have been resolved, and is further strengthened in the 4.3.2 release...

There was no sense in continuing the deployment on the old PHP base, when it was all going to torn out and upgraded...

We hope to have the PHP 4.3.2 upgrades announced within the next two weeks...

--
Terra
--what a long strange trip it's been--
FutureQuest

[Crow]

Well posting the news about it would be nice..
not w8ing till someone asks about it.

[Terra]

We are working on it, and it will be announced and released when it is ready for production...

Secure_Mode(TM) is an exclusive ground breaking and technological feat, and it is taking some time to strengthen and perfect it's overall operations... This is something that does not take kindly to being rushed or pushed out the door to satisfy the impatient...

Until then, I humbly request your patience so that we may give it the time and focus it deserves...

--
Terra
sysAdmin
FutureQuest

[Dr Mirth]

Hi Terra,

Okay, I'll admit I'm impatient. :o

But I'm just wanting to know if there is any new news to report regarding the Secure_Mode deployment? I'm just itchin' to get Gallery installed on my web site and start integrating it with my forums.

[Terra]

Okay, I'll admit I'm impatient.LOL, that makes about 5 of us then... ;)

-----
Since there was a large request for mbstring being added to PHP, we decided to go ahead and roll it in on our upcoming PHP 4.3.2 release... It has just come out of (site owner) Beta testing with good results and was pretty much the focal point for the delays...

From here, I have to start forging the production version for deployment... This is currently very high on my task list, with a good chunk of my weekend devoted to reworking the servers for it...

The PHP 4.3.2 deployment will also include installing our Secure_Mode™ to all servers...

As you can see, this is going to be a huge push forward and I'm hoping for a scheduling announcement next week with the actual deployment by next weekend...

--
Terra
--sometimes I feel like every milestone target with PHP 4.3.2 has been written on flash paper--
FutureQuest

[mromero]

Terra must be triple checking his servers to prepare for when the advanced users start to load them up with the "real stuff" he has denied us for so long ;-)

An updated version of Gallery is out. 1.3.4 - it appears to run faster than the previous version on my test account at opensourcehost.com

"Gallery v1.3.4 is both a new feature and bugfix release, and is recommended for all Gallery users. New features for v1.3.4 include: the ability to download your gallery to burn to CD or browse offline, additional photo print services, auto-rotation of JPEGs when possible, and the ability to add new customized description fields to photos. In addition, v.1.3.4 fixes numerous minor bugs, and extends support for PHP-Nuke to versions 6.5 and newer, and improves the Windows XP Publishing Wizard interface."

Terra's hacked version of PHP Safe Mode Off is running on several servers already - check the forums and you should see which servers they are.

Regards


8}

[Terra]

Terra must be triple checking his servers to prepare for when the advanced users start to load them up with the "real stuff" he has denied us for so long
Yes, that is a very real concern... There has already been noticeable resource usage increases on servers with Secure_Mode™ installed, since they have been loaded up with all sorts of real stuff... %)

I pretty much have been forced to use the equation: "Give an inch - They take a mile", with resource allocations and there has been quite a bit of exhaustive server reorganization to more effectively handle the more intensive sites... From what I can see our Secure_Mode™ ability is pretty darn popular and often it's limits are pushed beyond my wildest expectations...

--
Terra
--is nothing more than a glorified resource traffic cop--
FutureQuest

[phppete]

Hi Terra,

Are you aware of the bug with mbstring? I don't know if its fixed now but there was a major problem with mbstring and I believe PHP 4.1.2, mbstring was causing arrays and multi-dimensional arrays to lose the first character.

Are you aware of this? I know many hosts including my previous hosts, dropped mbstring due to this.

Thanks

Pete :)

[phppete]

me again :P

Just wondered what you guage as 'resource intense'. On previous hosts CPU idle has often hung around the 5 and 10% mark, and often drops to 0% CPU, personally I see this as an overloaded host. On the occasions I run TOP I don't see CPU idle below 90%, most times its 95%+plus.

So the question is what is acceptable CPU idle for FQ. (i'm using CPU idle rather than server load because I understand that is a better indicator of free resources)

Pete

[PaulKroll]

What sorts of scripts have been loaded up into the Secure_Mode[can't figure it out, tell me how to generate a superscript tm here please!] servers that are so different from the scripts that were loaded up under the old server config? Is it file creation/editing/access that's causing the load increase, or running of piped applications, or something else?
- Let's bet on the "well it's ALL of those..." response... :)

[phppete]

I thought Movable Type was the Master Resource Eater! ... I too would like to know whether there are heavy PHP scripts around or is it just that everyone has loaded everything and anything they can find just because they can?

Pete

[Terra]

I'll try to keep this short... :)

Is it file creation/editing/access that's causing the load increase, or running of piped applications,
Yes... Keep reading below...

I too would like to know whether there are heavy PHP scripts around or is it just that everyone has loaded everything and anything they can find just because they can?
Yes, PHP scripts can now fork/exec external programs to do it's work, such as the case is with ImageMagik, NetPBM, plus just about any other program you can think of like scripts forking out to pipe stuff through 'grep', 'perl', 'sed', 'awk', etc etc etc... It's pretty crazy out there what some of the stuff people can think up to do to a server on a lazy Saturday afternoon...

--
Terra
--has a wooden mallet with a lot of notches in it--
FutureQuest

[Terra]

So the question is what is acceptable CPU idle for FQ.
CPU idle is only one small indicator of many... Most of the time I go from the 'feel' of the server as each of our servers are very much unique in their essence of operation and personality... Some are heavy CPU, others heavy real memory, some use more swap than others, some have to manage a large number of forked processes, others are heavy Disk I/O, others are heavy external network I/O, others are heavy internal network I/O, etc, etc, etc and most are all a varying combination of the above...

ENIGMA is mostly internal network I/O + disk I/O (MySQL + logging), which shifts the CPU resources to the MySQL engines instead...

--
Terra
--there is a reason I like to name our servers--
FutureQuest

[Bob]

Originally posted by mromero:
An updated version of Gallery is out. 1.3.4 - it appears to run faster than the previous version on my test account at opensourcehost.com

"Gallery v1.3.4 is both a new feature and bugfix release, and is recommended for all Gallery users. New features for v1.3.4 include: the ability to download your gallery to burn to CD or browse offline, additional photo print services, auto-rotation of JPEGs when possible, and the ability to add new customized description fields to photos. In addition, v.1.3.4 fixes numerous minor bugs, and extends support for PHP-Nuke to versions 6.5 and newer, and improves the Windows XP Publishing Wizard interface." Can't comment as of yet on speed and new features BUT if you are upgrading 1.3.3 to 1.3.4 you will need to heed the following portions of the UPGRADE file:

"If a significant enough change has happened to Gallery, it will force you to go back to config mode and run through the config wizard again, but it will save all your old settings so that should be pretty easy."

"Once you've got the new code and have run the config wizard, you may be required to let Gallery upgrade all your albums."

As I have had to do both, One Album at a time including sub Albums...

Enjoy,
Bob

- Yes I am one of the Lucky ones ;) -