SkunkWEB - smell the power!
about .
docs
download
mailing list
news
CVS
SkunkDAV
PyDO
who
Search (Google):
+ -------------------------- +
| many thanks to SourceForge |
+ -------------------------- +

The SkunkWeb2 Web Application Server

Drew Csillag1

Abstract

This paper describes SkunkWeb, a web application server tool box written in Python[12]. In essence, it's merely a multi-process socket server. However when all glued together, SkunkWeb forms a robust, scalable, extendible and easy to use application server. The caching abilities make it an ideal application server for high volume websites as well as small websites on more meager hardware.

Why We Did It

Before I started writing SkunkWeb, I obviously examined the available open and non-open source application servers. I was generally dissatisfied, as the templating languages were either too constraining for the experts in the shop, too complicated for the non-experts, or the overall performance characteristics of the server were unacceptable.

Being a large web shop, we needed two major things from a web application server: a good templating language, and true enterprise abilities (vertical as well as horizontal scalability, reliability and robustness).

Since we couldn't find what we needed, we did what every other web shop on the planet would do... we did it ourselves.

Philosophy

The main Skunk philosophies are:
loose coupling
We should never force anyone (specifically ourselves) to take it all, but should allow people to take only the pieces they need to get their jobs done.
simplicity
Unnecessary complexity leads down the road to insanity. Do the simple thing unless you can truly justify a need to the contrary.

Goals

Being in the enterprise class, we needed four main things:
  1. speed
  2. speed
  3. speed
  4. usability

Process Model

Unlike the bulk of Python application servers (e.g. [14], [13] and [11]), we fork processes, not threads3. Application servers normally tend to be I/O bound, but if we can (and we can) reduce the I/O constraints (by caching and pushing client I/O into Apache), we'll be CPU bound (and we are). Being CPU bound, Python's threading model kills us since unless we write a pile of C code (which we wanted to avoid as much as possible) the Python interpreter will really only use one processor, not all of them due to Python's global interpreter lock4. By forking, SkunkWeb will utilize all processors it can get it's hands on.

Forking processes instead of threads makes a number of things simpler. Reliability is simplified because if a child dies (due to core dump or other abnormal exit), the parent will just spawn another. If it leaks resources (memory, file descriptors, etc.) there is a maximum request count so that the child will eventually commit suicide and release the resources. The kids can pretty much do whatever they want5 and the parent is the only thing able to crash the whole server. Since the parent is easy to take care of (it's mainloop is simple) the server will rarely go down, if ever6.

Fortunately or not, being a forking server means it doesn't run on Windoze (AFAIK). Update: a recent (3.4b3) version has been confirmed to run on Windows using the Cygwin toolkit.

The main things threading allows is general process resource pooling. Specifically cache and database connections, which can be had without threading. SkunkWeb caches many things to disk. In any decent read-as-Unix-like operating system, the OS disk cache should keep the real I/O to a minimum, effectively caching in memory7. The OS will also handle any locking issues involved8 normally requiring mutexes in a threaded environment.

The database connection pooling can be mitigated in a few ways. SkunkWeb can cache the database connection on a per process basis which is often enough. In the event you cannot afford one connection per process, you can use remote components to a dedicated ``database server'' servicing all of the database queries via remote component calls. It's a bit more work to do it that way, but it's likely to pay off. As it turns out, since you can cache database results using data components (see section 9) in the ``database server'', the database issue becomes less of an issue you may have originally thought it would. More on remote components later.

Communication

SkunkWeb can use either TCP or UNIX domain sockets for the Apache[1] to SkunkWeb communication. If you are going to run Apache on the same host that SkunkWeb will run, UNIX domain sockets have a number of advantages:
  1. They're faster than TCP connections.
  2. They're not accessible from outside the box, making security simpler.
  3. You don't have to worry about exhausting the ephemeral TCP port range on your box.
Given that (3) only happens in the largest of installations, (1) and (2) are good enough reasons for using UNIX sockets. If you are going to run Apache on other boxes, TCP is really the only choice, but mod_skunkweb can be told to fail over to another host if SkunkWeb becomes unavailable for some reason. This rarely happens, if ever. Really. Except when the whole machine goes away.

We presume Apache is the web server front-ending SkunkWeb, but as the Skunk philosophy would indicate, anything speaking the SkunkWeb protocol (it's quite easy) can be used as a front end. In fact, SkunkWeb includes an HTTP service (see section 10) that can be used in lieu of Apache when you don't need the speed or configurability an Apache installation provides.

Caching, Caching, Caching

A relatively easy way to speed up an application server is by caching. In SkunkWeb, just about everything is cached. We cache the compiled forms of STML templates, python code and message catalogs to both memory and disk, and we cache the output of components to disk.

The component output cache can get quite large since the size of the component output cache is proportional to the cross product of the number of cachable components and the number of different argument sets passed to them. One would ideally like to share the cache between application servers serving the same content. It is a relatively simple matter to put it on one or a number of NFS (or any other shared filesystem) and have SkunkWeb take care of it, and do the right thing if one of the NFS servers die.

Templating Language

Any serious web application server needs a templating language9. After shopping around a bit, I found something all of them had in common was... they stank. All of the ones I saw were either some variant of ASP[2] (e.g. JSP[5], PSP[10], etc.), or an emasculated tag-based language (e.g. DTML[8], CFML[3], etc.). The ASP variants have the problem of nonprogrammer HTML people doing bad things to the code10, or it gets hard to mentally separate the code from the markup. The existing tag based languages have the problem of not letting you do what you want, or having to do a lot of work to get it done.

My solution was this: write an extendable tag-based language that let you use strings and Python expressions as tag arguments anywhere (with a few exceptions). This insulates you from the amateurs, and still allows you to have ultimate power.

Since the language, STML, was tag based, all we really needed to do was:

  • Make regular Python code easy to load from STML, so you didn't have to do everything in STML.
  • Write an STML to Python byte-code compiler.

The first item was easy, add an import tag. This way you put your business logic in regular Python modules, and put the rendering logic in STML components. Doing it this way produced many more advantages than I expected.

  • HTML coders don't necessarily have to know all that much Python. STML is easy to learn, especially for people that haven't done much programming before.
  • If a more advanced HTML coder needs something built and doesn't want to wait for a Python module to be written by somebody else, they can do it themselves in either Python or STML.
  • If you are doing a lot of tag stuff in an STML, there is a ``calculated inefficiency'', pushing you to use Python modules to do some of the work.

The STML to Python byte-code compiler turned out to be a big win concerning performance. Before the compiler, STML was interpreted, which had these problems:

  • The parser was slow.
  • The interpreter was slow.
  • Using cPickle to cache the parse tree was only marginally faster than starting from scratch.

While compilation is slower than parsing, loading the compiled code from cache is roughly two orders of magnitude faster11 since the marshal module loads code objects really well. The fact we compile to Python byte code meant there was no separate STML interpreter anymore (just the regular Python VM) which sped things up even more.

Another important thing was to make STML extendable. I knew I didn't know all the tags we'd need and would have to add some. Creating a new tag is not too hard. It involves writing a class to handle the tag and inserting an instance of the class into a tag registry. Not needing to know how all of the compiler works to write a tag made it really cool.

Internationalization

Given that we are a tri-lingual content shop12, having to create a template for each language, or having a slew of ``if language is...'' statements was just dumb. We added message catalogs to SkunkWeb for this reason. This way one can write language independent templates. Message catalogs can also handle argument substitution. This was needed because the grammars of the three languages we use don't always put things you'd substitute in a message in the same relative locations (so normal string substitution was out).

Message catalogs (really just dictionaries of language to message name to message mappings) were probably the simplest thing added to SkunkWeb, but save a tremendous amount of work.


Components

Components in SkunkWeb are a distinct unit of work, similar to a function call in many regards. There are three kinds13 of components:
include
these take no arguments, run in the namespace of the calling component, are not cachable, and produce output
regular
these take arguments, can be cachable, run in their own namespace and produce output.
data
these take arguments, can be cachable, run in their own namespace and can return an arbitrary object.

Components can be written in either STML or Python. If you want a component call to be cachable, you call it with cache=yes14 and inside the component, you specify a cache lifetime by saying one of the following:

  • cache for some duration (e.g. 10 minutes, 3 hours, etc.)
  • cache until some time (e.g. quarter past the hour, every other Tuesday at 5pm, and anything you can express with one or more RelativeDateTime objects)

If you are wondering what you do with data components, try thinking of them as memoized function calls. If you call them with cache=yes and the same arguments before the cache expires, they return whatever they returned on a prior call. Thus, you can cache commonly used things, such as stock-tickers, database rows, latest news from slashdot.org, and anything else that cPickle can serialize.

You can also tell SkunkWeb to defer component execution and say, ``give me whatever is in the cache, even if it's expired (but not too old). If it is expired, evaluate the component after the response is sent back to the client''. You can even set up SkunkWeb so if component evaluation fails, it falls back to whatever is in the cache, even if expired.

SkunkWeb also has the ability to call components on other SkunkWeb servers (a.k.a. remote component calls). In truth, they're just remote procedure calls that happen to be cachable.


Extendibility

Extendibility in SkunkWeb is done by what we call ``Services''. Services are just plain Python modules which generally assume that the SkunkWeb framework is there and hook into it using, well, Hooks. A number of people have asked me why I went with hooks over doing some kind of subclassing OOP-y kind of thing. My answers are these:
  1. When hook calling order is important, it is much easier to manipulate a list than to get the subclass hierarchy correct. There is a reason the Apache people went the same way with Apache 2.
  2. Services may add hooks to the server other services want to use, making services compatible only with some set of hooks.
  3. Because I said so. I'm not big on class hierarchies when they'd just be single instances. Plus, simpler hook mechanisms are so much easier to trace (e.g. you can print the hook and see what functions it calls).

Database Connectivity

Database connectivity in SkunkWeb is really simple. All you really have to do is write a service15 that makes the connection to the database and stows it somewhere convenient for template and module authors. Since SkunkWeb itself doesn't use threads, writing these modules is incredibly simple, and they are often less than 50 lines (assuming of course that Python bindings for the database already exist).

Content Management

Content management on large web sites can be exceedingly difficult. Especially when you want to push the changes more or less atomically to the live servers. SkunkWeb includes par files to make this easier. Par files are similar in purpose to Java[4] jar files. Pushing is done by packaging up your components into a par file and instructing SkunkWeb to use the par files instead of (or before) the normal document root when fetching components. This way, you push all of your par files, restart the servers (there is a soft restart available by sending SIGHUP to the parent process) and the new content shows up all at once. No worrying about a ``half push''. Update: Par files are no longer supported as there is now support for Zip files and tar files.

Using par files turned out to have an additional benefit I hadn't thought of when I first added the feature. Par files are loaded at server start. When we go searching for templates, we just scan dictionaries in memory instead of using open and os.stat to retrieve the necessary info about the templates. Thus the system call load is significantly reduced.

Conclusion

SkunkWeb and its predecessors have been used in continuous production at StarMedia since 1998 and has proven over and over again to be a scalable, robust16 application server which its users have found better than the alternatives17.

Bibliography

1
Apache.
http://www.apache.org

2
Active Server Pages.
http://msdn.microsoft.com/workshop/server

3
CFML Language Reference.
http://www.allaire.com/documents/cf45docs/acrobatdocs/45langref.pdf

4
Java.
http://www.javasoft.com

5
JavaServer Pages.
http://java.sun.com/products/jsp

6
MySQL.
http://www.mysql.org

7
Oracle.
http://www.oracle.com

8
Michel Pelletier and Amos Latteier.
Zope DTML Reference.
http://www.zope.org/Members/michel/ZB/AppendixA.dtml

9
PostgreSQL.
http://www.postgresql.org

10
Python Server Pages.
http://www.ciobriefings.com/psp/

11
PyWX - Python for AOLserver.
http://pywx.idyll.org

12
Guido van Rossum.
Python.
http://www.python.org

13
Webware.
http://webware.sourceforge.net

14
Zope.
http://www.zope.org

About this document ...

The SkunkWeb2 Web Application Server

This document was generated using the LaTeX2HTML translator Version 2K.1beta (1.47)

Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The command line arguments were:
latex2html -split 0 SkunkWebIPC10

The translation was initiated by Drew Csillag on 2001-08-10

Footnotes

... Csillag1
drew_csillag@geocities.com
... SkunkWeb2
http://skunkweb.sourceforge.net
... threads3
though your modules can use threads with the caveat of many SkunkWeb API functions (notably callComponent) are not reentrant
... lock4
Using a Python interpreter not built with threading support will speed up SkunkWeb a bit
... want5
kids have it so easy these days... :)
... ever6
assuming the machine stays up, anyway
... memory7
although it's not as efficient as shared memory would be
... involved8
though SkunkWeb does not use locking of any kind (this includes file locking)
... language9
a programmer can't live on servlets alone :)
... code10
it was translated into Spanish on one notable occasion
... faster11
my torture test document went from 100ms/load to 4ms/load
... shop12
we'll try anything :)
... kinds13
well, mainly three, there are remote components also :)
...cache=yes14
or cache=force or cache=old or cache=defer
... service15
services for Oracle[7], PostgreSQL[9], and MySQL[6] are already available as part of the standard distribution
... robust16
and many other buzzwords
... alternatives17
``All app servers suck, SkunkWeb just sucks less''
Drew Csillag 2001-08-10

---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
(smell the power!)