Pregenerate pages: serverload & bandwidth...

[pier]

I'm currently working on a site on which each and every page is generated through datebase-access, ranging from simple selects to extensive searches, also changing over time (counters, ratings etc.).
On some part in time, it might become handy to pregenerate the pages, to reduce server-load and speed up page-access (no waiting time for query-executing).

Using mod_rewrite, wget, cron etc. it is easily to grab these pages each XX minutes/hours or so (depending upon which page), and redirect normal browsers to the fixed pages.

However, a few questions are left for me:
a) Is bandwidth from local accesses calculated in the monthly plan ? As it will go through apache, the accesses will probably appear in the logfiles (correct me if I'm wrong), but no bytes will ever leave the NOC, I hope. In worst case, every byte gets double-counted, as being outbound AND inbound traffic.
b) On which time-frame it is 'allowed' to fire of cron-jobs ? I have some pages that still will be entirely dynamic, while other might need to be updated once every 15 minutes. I can tell wget to be very relaxing (one page every x seconds).

Pier

[Terra]

a) Yes, if it's in the Apache raw logs - then it's bandwidth will be counted against you...
On a side note, you are not only paying for the bandwidth itself, but also the processing and memory usage that it takes for Apache to serve that request...  Resources are a finite source, and your usage of a resource, lowers someone elses ability to use it at the same time...  So in effect, we are recovering server cycle costs via bandwidth fees as well since we can't recover those cost directly...

b) For the 'heavy' jobs - keep them after 2:00am EST as the servers have already begun to ramp down...  For continuous checks, the minimum time periodicity should be 5 minutes...  Also, make note that the cron jobs are under new process controls - as the cron system was being *heavily* abused...  The last straw was when someones custom web analyzer had chewed up over 480Mb of RAM and just 'sit-n-spun' forever almost taking SIX down...

MAXIMUMS for cron spawned jobs
Runtime: 60 seconds
Memory: 32Mb
Proc: 32 (forks) *will be changing this to 8 soon
Open Files: 64 handles

If your 'heavy' programs work within those limits, then you should be fine...  If they won't - then you will need to rethink how you are trying to accomplish the desired result...

--
Terra
--Trying to balance the fine line of site owner freedom and server protective limits--
FutureQuest