Development Seed Blog

Speeding Up Aggregation in Drupal with a Daemon

Recently we’ve been working to get our team aggregator and media analyzer Managing News running faster – we want it to fly. To do this we’ve had to really push what Drupal and the LAMP stack can do. Aron and Alex have done great things with aggregation and feed parsing to extend the volume of content that can be collected, which is essential for Managing News since is can aggregate tens of thousands of articles every day. But we still wanted to be able to aggregate content faster.  

A major thing that slows down adding content to Managing News is its semantic analysis – every piece of content that the system pulls in is automatically browsed and given tags that describe it. To do this we mostly use third party web services like Yahoo's term extraction API. Waiting for Yahoo to process the text of each article coming in can add a significant amount to time to the process of adding content - it keeps cron running longer, which increases the chances of a bad cron run, and ties up the system’s resources to just wait. 

To get around this we looked for ways to move the semantic analysis elsewhere. We settled on the idea of using a small external program to talk to tagging web services and do it's own analysis of content in our Drupal system. We looked at a couple options, and quickly decided to write a Python daemon. (A daemon is a program that runs in the background on a server. It waits for certain events and then takes specified actions. In our case the daemon waits for new content, and then tags and processes it as it becomes available.)

We choose Python over php for two main reasons:

• We wanted to get away from the LAMP stack. Apache threads serving Drupal pages can consume upwards of 30 mb of memory. The Python script we've got uses less than 4 mb. Potentially we could run 7 or 8 instances of the Python script with the same memory footprint as a single Drupal page load.
• Daemonizing a process with Python is easy

Our current version consists of a Drupal module and Python scripts. The Drupal module serves three functions: 1) it queues up new nodes for the daemon to process, 2) it allows you to configure the daemon's behavior, 3) it creates tables inside the Drupal database that the daemon needs.

The Python programs consists of a very thin database wrapper, a minimal interface to Drupal for loading nodes and working with taxonomy terms, a daemonizing script, and finally a set of plugins to process nodes. Each time a new story from an RSS feed is added to the system, it’s put in a queue for the daemon. The daemon then grabs batches of nodes from this queue and processes them together, first giving the plugins a chance to act on each node individually and then as a batch.

Already this has been a great addition to Managing News. Aggregation is lighter and faster, and the system itself requires less resources – which means we can aggregate more. We've got lots of ideas on other things we can do with our little daemon and are very excited to see what we can use it for.

And yes, we do intend to release the code. But as total pythons newbs, we want to go over it with a fine toothed comb to make sure we're not doing anything too offensive to any Pythonistas hidden within the Drupal community :)