PDA

View Full Version : Pregenerate pages: serverload & bandwidth...


pier
01-26-2001, 02:12 PM
I'm currently working on a site on which each and every page is generated through datebase-access, ranging from simple selects to extensive searches, also changing over time (counters, ratings etc.).
On some part in time, it might become handy to pregenerate the pages, to reduce server-load and speed up page-access (no waiting time for query-executing).

Using mod_rewrite, wget, cron etc. it is easily to grab these pages each XX minutes/hours or so (depending upon which page), and redirect normal browsers to the fixed pages.

However, a few questions are left for me:
a) Is bandwidth from local accesses calculated in the monthly plan ? As it will go through apache, the accesses will probably appear in the logfiles (correct me if I'm wrong), but no bytes will ever leave the NOC, I hope. In worst case, every byte gets double-counted, as being outbound AND inbound traffic.
b) On which time-frame it is 'allowed' to fire of cron-jobs ? I have some pages that still will be entirely dynamic, while other might need to be updated once every 15 minutes. I can tell wget to be very relaxing (one page every x seconds).

Pier

Terra
01-26-2001, 04:41 PM
a) Yes, if it's in the Apache raw logs - then it's bandwidth will be counted against you...
On a side note, you are not only paying for the bandwidth itself, but also the processing and memory usage that it takes for Apache to serve that request...[nbsp][nbsp]Resources are a finite source, and your usage of a resource, lowers someone elses ability to use it at the same time...[nbsp][nbsp]So in effect, we are recovering server cycle costs via bandwidth fees as well since we can't recover those cost directly...

b) For the 'heavy' jobs - keep them after 2:00am EST as the servers have already begun to ramp down...[nbsp][nbsp]For continuous checks, the minimum time periodicity should be 5 minutes...[nbsp][nbsp]Also, make note that the cron jobs are under new process controls - as the cron system was being *heavily* abused...[nbsp][nbsp]The last straw was when someones custom web analyzer had chewed up over 480Mb of RAM and just 'sit-n-spun' forever almost taking SIX down...

MAXIMUMS for cron spawned jobs
Runtime: 60 seconds
Memory: 32Mb
Proc: 32 (forks) *will be changing this to 8 soon
Open Files: 64 handles

If your 'heavy' programs work within those limits, then you should be fine...[nbsp][nbsp]If they won't - then you will need to rethink how you are trying to accomplish the desired result...

--
Terra
--Trying to balance the fine line of site owner freedom and server protective limits--
FutureQuest