Blog: Aggregation

Talking with Bloggers, Campaigners, and Progressives about Tracking New and Traditional Media

A lot of campaigners in Washington, DC for Rootscamp just got a peak at Managing News - our news tracker and analyzer tool. If you missed the session, you can check out some videos of the tool here and contact us for a seven day free trial.

The session led into a great discussion on what people in the political realm need with a tool like this - particularly extreme media analyzers and bloggers. That there's a need to track online chatter on both a macro and a micro level wasn't in question - the people in the room knew that was necessary and it's been a recurring theme in all the sessions I've been to so far today. This in itself is really great to hear because we built Managing News to do just that, and to give a snapshot view of what’s happening online around a topic and make it easy for people to take action on it.

There was some good excitement about Managing News' ability to roll a lot of what people are already doing with Bloglines, Google alerts, and manual news reading into one shared platform, and that one that automatically adds information like keywords and locations to the news it sucks in. We also heard some great feature requests - like source exclusion, clipping reports printed in various formats, and close captioning tracking - that will be keeping us busy for awhile : ) Let us know if you have a specific news tracking or analyzing need. We're happy to talk about what Managing News and other tools can do to help.

Help Us Work on This New Tool

Managing News is a Drupal powered news tracker and analyzer that helps communications teams and PR firms monitor news and trends across the internet. We're looking for a few smart, creative engineers to join our tight knit team to work on this new tool. If you're interested in Drupal, the news, aggregation, and working on a totally different type of tool, check out the job description. Drop us a line if you want to learn more: jobs@managingnews.com.

A Migration Script to Make the Transition Easy

After a lot of hard work, we've released FeedAPI 1.0. But what does this mean for Leech users? Simply put, in most cases it means a better aggregation option is now available - and with the ready-to-use migration script explained below the transition is easy.

As most active users know, Leech isn't being further developed. On the other hand, FeedAPI is actively maintained and constantly improved. This alone means that in most cases, the obvious move is to migrate to FeedAPI. You probably don't want replace Leech if your site is running nicely, doesn't need new features, and has no performance problems. But if that isn’t the case, then migrating to FeedAPI is a good idea. If you're still in doubt, try FeedAPI out at feedapi.novaak.net. (Note: The site can be snail slow, but that's not because of FeedAPI - it's on shared hosting :) )

Follow these instructions to upgrade to FeedAPI from Leech:

Requirements: Leech 1.9 installed, MySQL, PHP4-5 (Note: The migration script does not work with PostgreSQL)

1) Download the migration script.

2) Go into maintenance mode offline (admin/settings/site-maintenance). Copy the leech2feedapi.php script to Drupal's main directory. This is the directory where Drupal's index.php can be found.

New Flexible Aggregator for Drupal Ready for Use

After being in development for about seven months, we released FeedAPI 1.0 nearly two weeks ago! This is really exciting for me and everyone else who has been craving a more flexible aggregator for Drupal.

We wanted FeedAPI to be as flexible and as fast as possible, and I'm happy to say that the end result has reached those goals. All feeds are represented by nodes. Feed items remain independent from the node system - you can choose, whether you would like to store them as nodes or fast flat database records. Per default, FeedAPI is configured to use SimplePie as a feed parser, which works perfectly in most cases. But if you have particular requirements, you can switch to the alternative Common Syndication parser or write your own.

A particularity nice thing about FeedAPI is that you can configure your feed processing preferences on a per content type basis. Cron processes are performance optimized and work on a time limit rather than a number of feeds limit - a big relief for tuning cron performance. FeedAPI also collects important statistics for monitoring wicked huge aggregating sites. Additionally, FeedAPI has a straightforward configuration and is easily extensible through an API. (Just to know: The content-type preset (among others) is entirely Alex's concept and work, and Jose did a lot to make FeedAPI faster.)

In the near future, we plan to further tune FeedAPI's performance and make it more feature-rich (as I type this blog entry, we are working on committing new features that aren't included in 1.0 :) )

For now, I'd like to share with you what you can do with FeedAPI 1.0:

• Install it on a PHP4 or PHP5 site with a MySQL or a PostgreSQL database
• Cache feed downloading to save bandwidth and time
• Turn feed items into nodes
• Turn feed items into lightweight items – use FeedAPI like the good old core aggregator
• Adjust cron processing of feeds based on elapsed time. No more php timeouts
• Get statistics about feed processing
• With Feed Element Mapper, map feed item elements to CCK fields or taxonomy
• Import your feeds from anywhere with OPML import
• Export OPML
• Inherit taxonomy and organic group settings from feeds to feed items
• Views support, having a predefined views for exploring feed items for a specific feed
• Content-type based configurable presets
• Promote items to the front page according to your settings
• 1-click feed creating block. Just supply the feed url and the feed is created
• Purge or not purge outdated feed items – it's your choice
• Automate tests (simpletest module yeay!) - easy checking for problems on your installation.
• Auto-detect feed URL, you do not have to know the RSS/ATOM link, you can just supply the site URL, for example http://drupal.org
• Access to a documented API, you can write parsers or processors and use FeedAPI’s power in feed management
• Ready-to-translate .po files in the package
• +1: low turnaround time issue queue-based support :D

A More Efficient Way of Keeping People Posted

As we have these two powerful frameworks, FeedAPI and the new Notifications framework I talked about  in my last post, I was thinking it would be cool to have them work together. Users already had the option to stay subscribed (i.e. to all nodes created of a given type), but this didn't seem like a very efficient way of getting notifications for all updated feed items. 

So I’ve implemented this new notifications plug-in that allows you to subscribe to a feed and then get lists of new/updated items. To see it working please take a look at the notifications_feed.module in the notifications package.

The resulting email notification looks like this:

This simple module takes advantage of the existing Notifications and Messaging frameworks to implement - with just a few lines of code - a new event type (Feed update) and a new subscription type (Subscribe to feed) with all the features, such as configurable sending methods, queueing, and custom message templates.

The idea, in short, is that we can now easily create any custom plug-in to keep users posted about anything with just a few lines of code, while having all the gory details like message templating and delivery, user subscriptions,  notifications queueing for large sites, etc. automatically handled by these modules. 

And you don't need to create any new table or field for managing all this data!

The notifications_feed.module is already part of the Notifications Framework.

How To Create A Mapper

For those of you who after reading my last post on the Feed Element Mapper are now interested in the actual works of the module, below is a step by step guide on how to build your own mapper.

But first, three things needed to happen before we were able to create the map-any-element-to-CCK feature:

• Jess from WETA poked me about it for their TV and Radio feeds at the Drupal Meetup in DC on Tuesday :)
• FeedAPI's parser_simplepie now passes on the raw feed item, and with that anything it detects on the feed, on feed_item->options->raw
• Feed Element Mapper offers a generic CCK mapper

From those three, the one I would like to explain a bit better is the third because it shows how easy it is to write your own mapper so you can map any feed item element identified by FeedAPI to virtually any property of a node. Let me explain how I built this mapper:

1)
This is the empty implementation of hook_feedapi_mapper(). It's just sitting there and doing nothing. In this case, it' the implementation for the content module, the hook starts with content_.

function content_feedapi_mapper($op, $node, $field_name, 
				$feed_element = array(), $sub_field = '') {

}

• $op tells us which operation is being executed ('describe', 'list,' or 'map').
• $node is the node object we're operating on.
• $field_name is the name of the field on the node we are being called with (e. g. 'field_severity' for $node->field_severity, or 'taxonomy' for $node->taxonomy).
• $feed_element is only set on $op == 'map' and contains the exact element that has been mapped to the given field. This element can be a string, a number or an array of string or numbers.
• $sub_field is advanced stuff - it contains a value only if you offer more than one mapping target on $op == 'list', which we won't do here (e. g. a taxonomy mapper offering a mapping for each vocabulary).