Blog: Feedapi

FeedAPI Mail Plugin Lets You Aggregate Your Email With Your News Feeds

Keeping up with all the information we get every day is no easy job. We follow hundreds of websites using feed readers and get hundreds of emails every day. If that wasn't enough, there are new sites everyday producing their own streams of information, sites like Flickr, Facebook, Twitter, and probably someone is launching a new one as I'm writing this. We definitely need help to manage all this information, so we use tools like email filters, feed aggregators and readers, and tagging tools.

Just one of the tools mentioned above means one more piece of software or website to deal with. And that means that when we find some other tool that can mix what two tools do into just one, we have at least some hope that the next day we'll have some spare time to do the real work. That's what FeedAPI Mail does for FeedAPI in Drupal -  adds one more functionality to the mix. 

Now we have the first proof-of-concept module for a FeedAPI plugin that adds mailing lists into the equation. With this plugin, you can follow mailing lists and enjoy the same features available for the other content. It treats individual mailing lists as if they were web feeds, allowing the same features for incoming emails as for the rest of the content, including automatic tagging (and geotagging), reading the emails in a single news reader, easy sharing with your team, and so on. Here's a look at it running in our team news aggregator/analyzer Managing News:

This FeedAPI Mail module works together with Mailhandler and FeedAPI, which grabs the rest of the feeds and presents them in a single web interface. In addition it has also a specific mail reader UI that can display threads and authors, which puts single emails in context.

To get started we need a mail account and to set up a mail inbox with Mailhandler so emails are read and made into nodes. Next we need to add a 'Mailing list' and set it up with the mailing list email address so incoming emails can be classified as coming from different lists. Then we just need to subscribe our email address to the mailing list (which may take some manual steps to handle confirmation emails), and we're all set.

What's to Come with FeedAPI Mail

Or, Check Out the Patch

This year’s Google Summer of Code season I’ve got the distinct pleasure of mentoring Aron Novak’s work for a new aggregator in Drupal 7. Aron’s well into his task and has just rolled a patch for core and an alpha 2 version – time to share why I think that this patch is important and why you should have a look at it. If you’re into aggregation and Drupal, that is.

Drupal’s original aggregator module was designed foremostly for pulling news feeds into your site and displaying in a straightforward fashion: no workflow, very basic permissioning, no API for interacting with feed items, no event aggregation, no custom parsers – to name a few limitations.

Soon contrib modules mushroomed that addressed one or the other shortcoming of the core aggregator: a list of them would start with the aggregator 2 module which was published in the fall of 2005 and would include Leech (I don’t regret its demise), Aggregation (first time use of SimpleXML for parsing in Drupal), SimpleFeed (first extensible architecture) and FeedAPI

Use FeedAPI, Feed Element Mapper, and Embedded Media Field to Aggregate Video from Blip.tv

Sean Effel from Drupal Therapy just put up a wonderful screencast on how to pull blip.tv videos from RSS feeds into Drupal. In the screencast, Sean shows step by step how to set up your Drupal  site with the FeedAPI, Feed Element Mapper, and Embedded Media Field modules to aggregate video feeds from blip.tv.

If you're interested in pulling video into your website or online app, you'll definitely want to check this out.

A Migration Script to Make the Transition Easy

After a lot of hard work, we've released FeedAPI 1.0. But what does this mean for Leech users? Simply put, in most cases it means a better aggregation option is now available - and with the ready-to-use migration script explained below the transition is easy.

As most active users know, Leech isn't being further developed. On the other hand, FeedAPI is actively maintained and constantly improved. This alone means that in most cases, the obvious move is to migrate to FeedAPI. You probably don't want replace Leech if your site is running nicely, doesn't need new features, and has no performance problems. But if that isn’t the case, then migrating to FeedAPI is a good idea. If you're still in doubt, try FeedAPI out at feedapi.novaak.net. (Note: The site can be snail slow, but that's not because of FeedAPI - it's on shared hosting :) )

Follow these instructions to upgrade to FeedAPI from Leech:

Requirements: Leech 1.9 installed, MySQL, PHP4-5 (Note: The migration script does not work with PostgreSQL)

1) Download the migration script.

2) Go into maintenance mode offline (admin/settings/site-maintenance). Copy the leech2feedapi.php script to Drupal's main directory. This is the directory where Drupal's index.php can be found.

New Flexible Aggregator for Drupal Ready for Use

After being in development for about seven months, we released FeedAPI 1.0 nearly two weeks ago! This is really exciting for me and everyone else who has been craving a more flexible aggregator for Drupal.

We wanted FeedAPI to be as flexible and as fast as possible, and I'm happy to say that the end result has reached those goals. All feeds are represented by nodes. Feed items remain independent from the node system - you can choose, whether you would like to store them as nodes or fast flat database records. Per default, FeedAPI is configured to use SimplePie as a feed parser, which works perfectly in most cases. But if you have particular requirements, you can switch to the alternative Common Syndication parser or write your own.

A particularity nice thing about FeedAPI is that you can configure your feed processing preferences on a per content type basis. Cron processes are performance optimized and work on a time limit rather than a number of feeds limit - a big relief for tuning cron performance. FeedAPI also collects important statistics for monitoring wicked huge aggregating sites. Additionally, FeedAPI has a straightforward configuration and is easily extensible through an API. (Just to know: The content-type preset (among others) is entirely Alex's concept and work, and Jose did a lot to make FeedAPI faster.)

In the near future, we plan to further tune FeedAPI's performance and make it more feature-rich (as I type this blog entry, we are working on committing new features that aren't included in 1.0 :) )

For now, I'd like to share with you what you can do with FeedAPI 1.0:

• Install it on a PHP4 or PHP5 site with a MySQL or a PostgreSQL database
• Cache feed downloading to save bandwidth and time
• Turn feed items into nodes
• Turn feed items into lightweight items – use FeedAPI like the good old core aggregator
• Adjust cron processing of feeds based on elapsed time. No more php timeouts
• Get statistics about feed processing
• With Feed Element Mapper, map feed item elements to CCK fields or taxonomy
• Import your feeds from anywhere with OPML import
• Export OPML
• Inherit taxonomy and organic group settings from feeds to feed items
• Views support, having a predefined views for exploring feed items for a specific feed
• Content-type based configurable presets
• Promote items to the front page according to your settings
• 1-click feed creating block. Just supply the feed url and the feed is created
• Purge or not purge outdated feed items – it's your choice
• Automate tests (simpletest module yeay!) - easy checking for problems on your installation.
• Auto-detect feed URL, you do not have to know the RSS/ATOM link, you can just supply the site URL, for example http://drupal.org
• Access to a documented API, you can write parsers or processors and use FeedAPI’s power in feed management
• Ready-to-translate .po files in the package
• +1: low turnaround time issue queue-based support :D

A More Efficient Way of Keeping People Posted

As we have these two powerful frameworks, FeedAPI and the new Notifications framework I talked about  in my last post, I was thinking it would be cool to have them work together. Users already had the option to stay subscribed (i.e. to all nodes created of a given type), but this didn't seem like a very efficient way of getting notifications for all updated feed items. 

So I’ve implemented this new notifications plug-in that allows you to subscribe to a feed and then get lists of new/updated items. To see it working please take a look at the notifications_feed.module in the notifications package.

The resulting email notification looks like this:

This simple module takes advantage of the existing Notifications and Messaging frameworks to implement - with just a few lines of code - a new event type (Feed update) and a new subscription type (Subscribe to feed) with all the features, such as configurable sending methods, queueing, and custom message templates.

The idea, in short, is that we can now easily create any custom plug-in to keep users posted about anything with just a few lines of code, while having all the gory details like message templating and delivery, user subscriptions,  notifications queueing for large sites, etc. automatically handled by these modules. 

And you don't need to create any new table or field for managing all this data!

The notifications_feed.module is already part of the Notifications Framework.