The SkunkWeb2 Web Application Server
This paper describes SkunkWeb, a web application server tool box
written in Python. In essence, it's merely a
multi-process socket server. However when all glued together,
SkunkWeb forms a robust, scalable, extendible and easy to use
application server. The caching abilities make it an ideal
application server for high volume websites as well as small websites
on more meager hardware.
Before I started writing SkunkWeb, I obviously examined the
available open and non-open source application servers. I was
generally dissatisfied, as the templating languages
were either too constraining for the experts in the shop, too
complicated for the non-experts, or the overall performance
characteristics of the server were unacceptable.
Being a large web shop, we needed two major things from a web
application server: a good templating language, and true enterprise
abilities (vertical as well as horizontal scalability, reliability
Since we couldn't find what we needed, we did what every other web
shop on the planet would do... we did it ourselves.
The main Skunk philosophies are:
- loose coupling
- We should never force anyone (specifically
ourselves) to take it all, but should allow people to take only the
pieces they need to get their jobs done.
- Unnecessary complexity leads down the road to
insanity. Do the simple thing unless you can truly justify a need to
Being in the enterprise class, we needed four main things:
Unlike the bulk of Python application servers (e.g. ,  and
), we fork processes, not threads3.
Application servers normally tend to be I/O bound, but if we can (and
we can) reduce the I/O constraints (by caching and pushing client I/O
into Apache), we'll be CPU bound (and we are). Being CPU bound,
Python's threading model kills us since unless we write a pile of C
code (which we wanted to avoid as much as possible) the Python
interpreter will really only use one processor, not all of them due to
Python's global interpreter lock4. By forking, SkunkWeb will utilize all processors it can get
it's hands on.
Forking processes instead of threads makes a number of things simpler.
Reliability is simplified because if a child dies (due to core dump or
other abnormal exit), the parent will just spawn another. If it leaks
resources (memory, file descriptors, etc.) there is a maximum request
count so that the child will eventually commit suicide and release the
resources. The kids can pretty much do whatever they
want5 and the parent is
the only thing able to crash the whole server. Since the parent is
easy to take care of (it's mainloop is simple) the server will rarely
go down, if ever6.
Fortunately or not, being a forking server means it doesn't run on
Windoze (AFAIK). Update: a recent (3.4b3) version has been confirmed
to run on Windows using the Cygwin toolkit.
The main things threading allows is general process resource pooling.
Specifically cache and database connections, which can be had without
threading. SkunkWeb caches many things to disk. In any decent
read-as-Unix-like operating system, the OS disk cache should keep
the real I/O to a minimum, effectively caching in
memory7. The OS will also handle any locking issues
involved8 normally requiring mutexes in a
The database connection pooling can be mitigated in a few ways.
SkunkWeb can cache the database connection on a per process basis
which is often enough. In the event you cannot afford one connection
per process, you can use remote components to a dedicated ``database
server'' servicing all of the database queries via remote component
calls. It's a bit more work to do it that way, but it's likely to pay
off. As it turns out, since you can cache database results using data
components (see section 9) in the ``database server'',
the database issue becomes less of an issue you may have originally
thought it would. More on remote components later.
SkunkWeb can use either TCP or UNIX domain sockets for the
Apache to SkunkWeb communication. If you are going to run
Apache on the same host that SkunkWeb will run, UNIX domain
sockets have a number of advantages:
Given that (3) only happens in the largest of installations, (1) and
(2) are good enough reasons for using UNIX sockets. If you are going
to run Apache on other boxes, TCP is really the only choice, but
mod_skunkweb can be told to fail over to another host if
SkunkWeb becomes unavailable for some reason. This rarely happens, if
ever. Really. Except when the whole machine goes away.
- They're faster than TCP connections.
- They're not accessible from outside the box, making security
- You don't have to worry about exhausting the ephemeral TCP port
range on your box.
We presume Apache is the web server front-ending SkunkWeb, but as the
Skunk philosophy would indicate, anything speaking the SkunkWeb
protocol (it's quite easy) can be used as a front end. In fact,
SkunkWeb includes an HTTP service (see section 10) that
can be used in lieu of Apache when you don't need the speed or
configurability an Apache installation provides.
A relatively easy way to speed up an application server is by
caching. In SkunkWeb, just about everything is
cached. We cache the compiled forms of STML templates, python code
and message catalogs to both memory and disk, and we cache the output
of components to disk.
The component output cache can get quite large since the size of the
component output cache is proportional to the cross product of the
number of cachable components and the number of different argument
sets passed to them. One would ideally like to share the cache between
application servers serving the same content. It is a relatively
simple matter to put it on one or a number of NFS (or any other shared
filesystem) and have SkunkWeb take care of it, and do the right thing
if one of the NFS servers die.
Any serious web application server needs a templating
After shopping around a bit, I found something all of them had in
common was... they stank. All of the ones I saw were either some
variant of ASP (e.g. JSP, PSP, etc.), or
an emasculated tag-based language (e.g. DTML,
CFML, etc.). The ASP variants have the problem of
nonprogrammer HTML people doing bad things to the
or it gets hard to mentally separate the code from the markup.
The existing tag based languages have the problem of not
letting you do what you want, or having to do a lot of work to get it
My solution was this: write an extendable tag-based language that let
you use strings and Python expressions as tag arguments
anywhere (with a few exceptions). This insulates you from the
amateurs, and still allows you to have ultimate power.
Since the language, STML, was tag based, all we really needed to do
- Make regular Python code easy to load from STML, so you didn't
have to do everything in STML.
- Write an STML to Python byte-code compiler.
The first item was easy, add an import tag. This way you put your
business logic in regular Python modules, and put the rendering logic
in STML components. Doing it this way produced many more advantages
than I expected.
- HTML coders don't necessarily have to know all that much Python.
STML is easy to learn, especially for people that haven't
done much programming before.
- If a more advanced HTML coder needs something built and doesn't want
to wait for a Python module to be written by somebody else, they can
do it themselves in either Python or STML.
- If you are doing a lot of tag stuff in an STML, there is a
``calculated inefficiency'', pushing you to use Python modules to do
some of the work.
The STML to Python byte-code compiler turned out to be a big win concerning
performance. Before the compiler, STML was
interpreted, which had these problems:
- The parser was slow.
- The interpreter was slow.
- Using cPickle to cache the parse tree was only
marginally faster than starting from scratch.
While compilation is slower than parsing, loading the compiled code
from cache is roughly two orders of magnitude faster11 since the
marshal module loads code objects really well. The
fact we compile to Python byte code meant there was no
separate STML interpreter anymore (just the regular Python VM) which
sped things up even more.
Another important thing was to make STML extendable. I knew I didn't
know all the tags we'd need and would have to add some. Creating a
new tag is not too hard. It involves writing a class to handle the
tag and inserting an instance of the class into a tag registry. Not
needing to know how all of the compiler works to write a tag
made it really cool.
Given that we are a tri-lingual content shop12, having to create a template for each language, or having
a slew of ``if language is...'' statements was just dumb. We added
message catalogs to SkunkWeb for this reason. This way one can write
language independent templates. Message catalogs can also handle
argument substitution. This was needed because the grammars of the
three languages we use don't always put things you'd substitute in a
message in the same relative locations (so normal string substitution
Message catalogs (really just dictionaries of language to message
name to message mappings) were probably the simplest thing
added to SkunkWeb, but save a tremendous amount of work.
Components in SkunkWeb are a distinct unit of work, similar to a
function call in many regards. There are three kinds13 of components:
- these take no arguments, run in the namespace of the
calling component, are not cachable, and produce output
- these take arguments, can be cachable, run in their own
namespace and produce output.
- these take arguments, can be cachable, run in their own
namespace and can return an arbitrary object.
Components can be written in either STML or Python. If you want a
component call to be cachable, you call it with cache=yes14 and
inside the component, you specify a cache lifetime by saying one of
- cache for some duration (e.g. 10 minutes, 3 hours, etc.)
- cache until some time (e.g. quarter past the hour, every other
Tuesday at 5pm, and anything you can express with one or more
If you are wondering what you do with data components, try thinking of them
as memoized function calls. If you call them with cache=yes
and the same arguments before the cache expires, they return
whatever they returned on a prior call. Thus, you can cache commonly used
things, such as stock-tickers, database rows, latest news from
slashdot.org, and anything else that cPickle can
You can also tell SkunkWeb to defer component execution and say,
``give me whatever is in the cache, even if it's expired (but not too
old). If it is expired, evaluate the component after the response
is sent back to the client''. You can even set up SkunkWeb so if
component evaluation fails, it falls back to whatever is in the cache,
even if expired.
SkunkWeb also has the ability to call components on other SkunkWeb
servers (a.k.a. remote component calls). In truth, they're just
remote procedure calls that happen to be cachable.
Extendibility in SkunkWeb is done by what we call ``Services''.
Services are just plain Python modules which generally assume that the
SkunkWeb framework is there and hook into it using, well,
Hooks. A number of people have asked me why I went with
hooks over doing some kind of subclassing OOP-y kind of thing. My
answers are these:
- When hook calling order is important, it is much easier to
manipulate a list than to get the subclass hierarchy correct. There
is a reason the Apache people went the same way with Apache 2.
- Services may add hooks to the server other services want to
use, making services compatible only with some set of hooks.
- Because I said so. I'm not big on class hierarchies when
they'd just be single instances. Plus, simpler hook mechanisms are so
much easier to trace (e.g. you can print the hook and see what
functions it calls).
Database connectivity in SkunkWeb is really simple. All you really
have to do is write a service15 that makes
the connection to the database and stows it somewhere convenient for
template and module authors. Since SkunkWeb itself doesn't use
threads, writing these modules is incredibly simple, and they are
often less than 50 lines (assuming of course that Python bindings for
the database already exist).
Content management on large web sites can be exceedingly difficult.
Especially when you want to push the changes more or less atomically
to the live servers. SkunkWeb includes par files to make this
easier. Par files are similar in purpose to Java jar
files. Pushing is done by packaging up your components into a par file and
instructing SkunkWeb to use the par files instead of (or before) the
normal document root when fetching components. This way, you push all
of your par files, restart the servers (there is a soft restart
available by sending SIGHUP to the parent process)
and the new content shows up all at once. No worrying about a ``half
Update: Par files are no longer supported as there is now support
for Zip files and tar files.
Using par files turned out to have an additional benefit I hadn't
thought of when I first added the feature. Par files are loaded at
server start. When we go searching for templates, we just scan
dictionaries in memory instead of using open and
os.stat to retrieve the necessary info about the templates.
Thus the system call load is significantly reduced.
SkunkWeb and its predecessors have been used in continuous production
at StarMedia since 1998 and has proven over and over again to
be a scalable, robust16 application
server which its users have found better than the
Active Server Pages.
CFML Language Reference.
Michel Pelletier and Amos Latteier.
Zope DTML Reference.
Python Server Pages.
PyWX - Python for AOLserver.
Guido van Rossum.
The SkunkWeb2 Web Application Server
This document was generated using the
LaTeX2HTML translator Version 2K.1beta (1.47)
Copyright © 1993, 1994, 1995, 1996,
Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999,
Mathematics Department, Macquarie University, Sydney.
The command line arguments were:
latex2html -split 0 SkunkWebIPC10
The translation was initiated by Drew Csillag on 2001-08-10
- ... Csillag1
- ... SkunkWeb2
- ... threads3
- though your
modules can use threads with the caveat of many SkunkWeb API
functions (notably callComponent) are not reentrant
- ... lock4
- Using a Python interpreter
not built with threading support will speed up SkunkWeb
- kids have it so easy these days... :)
- ... ever6
- assuming the machine stays up, anyway
- although it's not as efficient as shared memory would
- though SkunkWeb does not use locking of any
kind (this includes file locking)
- a programmer can't live on servlets alone :)
- it was translated into Spanish on one notable occasion
- ... faster11
torture test document went from 100ms/load to 4ms/load
- ... shop12
- we'll try
- ... kinds13
- well, mainly three, there are remote components also :)
- or cache=force or cache=old or cache=defer
- ... service15
- services for
Oracle, PostgreSQL, and MySQL
are already available as part of the standard distribution
- ... robust16
- and many other buzzwords
- ``All app servers suck, SkunkWeb just sucks
---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----