1 Caching Architecture

The caching architecture of SkunkWeb is fundamentally rather simple from the OS's point of view and luckily doesn't get that much more complicated when you get to the upper level details.

For component output caching, which makes up the majority of the number of files in the cache directory, we take the output of the component and a hex string representation of the MD5 hash of the component arguments and write it to a temp file. We then rename it to the real file name. This is done so we don't have to worry about write atomicity and having the SkunkWeb load a half-written file, not to mention that this scheme works properly over NFS!

An example: say we have a component whose path is /foo/bar/baz.comp and the argument hash is 0123456789ABCDEFFEDCBA9876543210. The temp file will be written into a file named /foo/bar/baz.comp/01/23/hostname_pid.countervalue.tmp and subsequently renamed (when writing is complete) to /foo/bar/baz.comp/01/23/0123456789ABCDEFFEDCBA9876543210.cache in the appropriate component cache directory. There is another file written for component output with the same name as the cache file, but replacing .cache with .key which contains a somewhat textual form of the component arguments suitable for use when doing cache clears (see section 10.1.3, page ).

For compiled templates and compiled python code, there is no hash (and thus, no .key file), so the cache filename is path_to/templatenamec and the rest of the process is the same.