Comment Aggregation: Tightly Integrating Third Party Social Networks with Your Site
Pulling Comments from Flickr, Youtube, and Vimeo
For the Knight Pulse website we recently built, we came up with a simple way of embedding video and comments from Vimeo in a blog post. Here's how simple it is. Just copy and paste the URL of the video to the node form, submit it, and you're done.
Watch this screencast to take a look at the UI:
This feature is of great value because it creates a bridge between each video's two presences - the conversations happening on Knight Pulse and the conversations on Vimeo. In this post, I'll show you how we built the feature and how you can do it yourself for Vimeo and also other social networks like Flickr, YouTube, and, with a little bit of homework, almost any other site.
Embedding the Video
The first part is simple. Embedded Media Field (awesomely) supports a number of media sites and conveniently enough one of them is Vimeo. Install and configure CCK and Embedded Media Field for a content type and then we're ready to embed video just by copying/pasting the URL from the video's page.
Act 2: Pulling in Comments
This is the part we wrote from ground up. FeedAPI Comments aggregates comments to either Drupal comments or nodes. In our case, we would like to pull in comments right on the node that's embedding the video. That means we enable FeedAPI on this node's content type and enable the Comments parser (part of FeedAPI Comments) and the FeedAPI Comments itself.
Act 3: The Glue
At this point, we're nearly there. Now write a blog post, copy the URL, paste it to the Embedded Media field, paste to the FeedAPI URL field, and we're done. Wait, so why do we need to paste it twice? Well, because there are two fields with the same value.
So let's hide one and copy the value from the exposed one. We whipped up a glue module that hides the FeedAPI fields if an EMField is present and populates the hidden field with the EMField value on validation. The module is very simple and has no configuration. It just hides all FeedAPI fields if an emfield is present. For convenience sake, it ships with FeedAPI Comments, its name is "FeedAPI Comments - Emfield integration".
Behind the Scenes
The most exciting part for me is how simple it is to write comment parser integrations for new sites. The FeedAPI Comments parser is an API for defining page scrapers. For example the information for scraping the comments from flickr.com consists of a series of XPath definitions:
// start site-specific data
$post_vars = ''; // if the comments are available after a POST request
// Currently one path can contain one placeholder, no more (imagine, one placeholder - running value - one more loop inside loops!
$xpath_comments = array(
'//div[@id=\'DiscussPhoto\']/table/tr[%place]',
);
// No placeholder support in data members yet
$xpath_data_members = array(
'title' => '',
'body' => '//td[@class=\'Said\' and position()=2]/p/text()[1]',
'author' => '/html/body/tr/td[2]/h4/a',
'author_picture' => '/html/body/tr/td[1]/a/img/@src',
'date' => '//p/small',
'original_url' => '',
);
$placeholders = array(
'%place' => array(1, UNLIMITED),
);
// To decide if the given video feed is okay for this handler.
// preg_match, so perl-style please
$url_checker_regexp = '/flickr.com\/photos\//';If you know XPath, you can write a comment scraper definition for a new site within an hour. This API is a pretty effective concept and could be easily generalized to any scraping job - not just comments.
Props to the great work done by the maintainers of emfield and to Aron Novak, who wrote FeedAPI Comments module.
If you would like to try comment aggregation yourself, be sure to you use FeedAPI Comments 1.0 and the latest development version of FeedAPI.
9 Comments
I’m using FeedAPI as the
I’m using FeedAPI as the engine for a mashup and am exploring the FeedAPI comments module for a new aspect of it. I hope to get your take on whether it’s the right tool for the job:
Using FeedAPI, I have a feed of blog posts coming in and am creating nodes out of each blog post, using the Feed Item content type.
I would like to enable FeedAPI comments on the Feed Item content type, so that off-site comments to each blog post are automatically pulled in and posted as local-comments at our site.
Provided we use Xpath properly to make the comment-scraping work, is FeedAPI comments the right tool for the job? In other words, will it automatically pull in comments for all blog posts we’re pulling in? And will it check for new off-site comments to those blog posts upon each cron job?
Thanks either way, great addition to the FeedAPI suite.
Using drupal 5 or 6?
This is quite an exciting project!
Seeing that the FeedAPI add-on has a release for only Drupal 6, I’m assuming you are running the FeedAPI and Embedded Media Field combo on Drupal 6. Although Embedded Media Field recommends using Drupal 5.
Website Development
Its great, I will check the scraping part of the code.
scraper
This is nice, I will check the scraping part of the code, because I used the DataMiner API module for similar purpose, but you may have something better..
About the usability of the concept in the video. I don’t know. Scraping and mixing comments from 2 sites and allowing reply on the comments on my site. It does not seems as a good idea for me, as the original authors will not get the reply.. But I am sure there are other use cases for this functionality..
Addressing usability concerns
Pasqualle, you’re spot on with your concerns. That’s why on the site where we did comment aggregation for the first time we split out the aggregated comments and showed them separately from the Drupal comments: http://www.knightpulse.org/blog/08/11/19/have-twenty-minutes-mobile-phon…
To do so, we aggregated comments not as Drupal comments, but as nodes. Then we created a view of these nodes and attached it to the node with the embedded video.
FeedAPI Geo
I love all the work you guys are putting into the FeedAPI and I’m a grateful user of the module and its submodules. Are there any plans to release a dev version of the FeedAPI Geo module any time soon?
FeedAPI Geo?
Hi Wim,
I am not aware of any FeedAPI geo module – that doesn’t mean that somebody out there isn’t working on one though.
Feed Element Mapper beta 3 supports location module stype locations as targets, maybe this is what you’re referring to – http://drupal.org/project/feedapi_mapper
FeedAPI Geo
Hi Alex,
Development Seed’s very own Young Hahn talks about the in-house developed FeedAPI Geo module in his article News Tastes Better When Mapped:
FeedAPI Geo: A simple FeedAPI plugin I hacked together to tag a geocoded item with its respective country (in development, to be released soon)
So I was just wondering how that module was coming along :)
GeoNames
Wim,
The News Tastes Better When Mapped post is from 10 months ago. Against expectations, FeedAPI Geo never solidified into something beyond a custom module since then.
The basic recipe for it was very simple though: We had random lat/lon coordinates and wanted to convert them into the lat/lon coordinates of the countries they are in.
We used the geo names service through the geo names module to convert coordinates to country in a first step and then looked up the coordinates of the country in a second step.
Alex