shtml pages and caching

[nemesis]

So I found this new "toy" here at http://www.ircache.net/cgi-bin/cacheability.py -- basically, it reads your headers and tells you how caches are likely to handly your page.

I was under the impression that .shtml pages cache just like an .html page. However, the results I get for all of my .shtml pages read like this:

http://www.buffyguide.com/new/episodes.shtml
Expires[nbsp][nbsp][nbsp][nbsp] -[nbsp][nbsp]
Cache-Control[nbsp][nbsp][nbsp][nbsp] -[nbsp][nbsp]
Last-Modified[nbsp][nbsp][nbsp][nbsp] -[nbsp][nbsp]
ETag[nbsp][nbsp][nbsp][nbsp] -[nbsp][nbsp]
Content-Length[nbsp][nbsp][nbsp][nbsp] - (actual size: 50546)[nbsp][nbsp]
Server[nbsp][nbsp] Apache/1.3.9 (Unix) mod_macro/1.1.1 PHP/3.0.15[nbsp][nbsp]

This object will be considered stale, because it doesn't have any freshness information assigned. It doesn't have a validator present. It doesn't have a Content-Length header present, so it can't be used in a HTTP/1.0 persistent connection.
And the .html pages return what seems to be a much better result:
http://www.buffyguide.com/episodes.html
Expires[nbsp][nbsp][nbsp][nbsp] -[nbsp][nbsp]
Cache-Control[nbsp][nbsp][nbsp][nbsp] -[nbsp][nbsp]
Last-Modified[nbsp][nbsp] 1 week 2 days ago[nbsp][nbsp](Wed, 22 Mar 2000 04:27:10 GMT) validated[nbsp][nbsp]
ETag[nbsp][nbsp] "fe8bc-5ad0-38d84b9e"
Content-Length[nbsp][nbsp] 22.7K (23248)[nbsp][nbsp]
Server[nbsp][nbsp] Apache/1.3.9 (Unix) mod_macro/1.1.1 PHP/3.0.15[nbsp][nbsp]

This object doesn't have any explicit freshness information set, so a cache may use Last-Modified to determine how fresh it is with an adaptive TTL (at this time, it could be, depending on the adaptive percent used, considered fresh for: 1 day 23 hr (20%), 4 days 22 hr (50%), 1 week 2 days (100%)). It can be validated with Last-Modified.

So, according to this thing anyway, my .shtml pages will not be caching. Why? Is there a way to make sure the Last Modified date is set? Do I need to use the mod_expires headers (which I toyed around with, but decided I'd rather not)?

------------------
---------------------------------
The Complete Buffy Episode Guide
www.buffyguide.com (http://www.buffyguide.com)

[PaulKroll]

Basically, an shtml is dynamic content: apache doesn't know if it should return the date of the root file, or an included file, or if there'll be some CGI within, so it doesn't return Last Modified and such by default. Check out http://www.apache.org/docs/mod/mod_include.html#xbithack which gives you one way of setting the Last Modified (to the .shtml page's LM date) so the pages will be at least partially cachable.

[nemesis]

Went there, checked it out, added:

XBitHack full

and ran a couple of pages through the query, but still no dice. Images get a last-modified date, but not .shtml pages. Did I miss something? :(
------------------
---------------------------------
The Complete Buffy Episode Guide
www.buffyguide.com (http://www.buffyguide.com)

[nemesis]

Can someone please help (Terra, Justin?)? We just checked our stats and now that our whole site (more or less) is shtml, we have ZERO 304's (in relation to pages; images cache fine). How can we get them to at least send a Last-Modified and/or ETag header? I know that for shtml -- how is it supposed to know of which included thing to use as the Last-Modified date, but surely there is some way to set this. That XBit thing didn't help, unless we did something wrong. It seems that if it weren't possible to cache the pages, no one would use includes... Any ideas? Perhaps we're missing something incredibly obvious.

(hint, hint, nudge, nudge: any new news about the HCs or DSs? We're really beginning to need the bandwidth, and we so very much don't want to leave you guys.)
[This message has been edited by nemesis (edited 04-06-00@08:33 am)]

[Justin]

You might find this thread helpful - I do not believe the Xbit hack will work on our servers (but I'm not positive), but using </font><font face="Courier" size="3">Header add</font><font face="Verdana, Arial" size="2"> does in fact work (and is used on this UBB):

http://www.aota.net/ubb/Forum3/HTML/000922-1.html

Hope this helps.

------------------
Justin Nelson
FutureQuest (http://www.FutureQuest.net/index.php) Support

[nemesis]

You know I did read that thread before, but since it seemed to be explaining the exact *opposite* of what I wanted to do, I kind of skimmed past it. ;)

Yes, I was thinking about using an ExpiresByType in my .htaccess, but I couldn't come up with exactly what I would want to set it to. Basically, I want the pages to cache at least for a typical users' session, which I am going to assume is probably about an hour. (Hey, this *my* typical user I'm talking about, not reality's.)

So would I want to use ExpiresByType text/html A3600. Or is this actually too short of a time? Would I want it to be A43200 to cache them for 12 hours since they last accessed the page? It seems like I definitely don't want to go by modified date. Or is there actually a reason I would want to?

Don't I run the risk, though, when setting this, that even after the expiration date goes by they will still view a cached copy?

I was thinking about all this a couple of weeks ago and it started hurting my head, so I stopped.

[Justin]

I honestly do not know the best method (or if there even is a &quot;best&quot; method) to do this - every time I think about caching my head hurts too...

The problem is that the caching depends on:

1) Web server (what it sends header-wise)
2) ISP (if they're running a cache)
3) Browser (make, model, version, build, platform, etc)
4) User settings (cache/history/paranoia)

And of course the alignments of the planets, altitude/air pressure, and so on. The fact is, there is never a way to make sure that a page does or does not get cached, much less trying to specify a period of time to cache a page...

I've had the best luck simply using PHP pages, mod_rewritten to .html pages, and setting no-cache headers for those pages which should not be cached and just hoping that the .html extention (which is spoofed) will cause those pages to be cached when appropriate. Generally they remain cached for me until I close the browser... but I'm close to sea level ;)

Hope this babbling helps in some way... or at least doesn't cause you to give up :)

------------------
Justin Nelson
FutureQuest (http://www.FutureQuest.net/index.php) Support