It’s been some time since we released the Messaging and Notifications, and the modules are quite healthy. Besides us using them in our site builds and intranet, they’re already running in more than 3,000 other sites, according to the Drupal.org usage statistics report. If we needed more encouragement to keep working on them, there it is :-)

We’re planning a big release for both frameworks that will include new features, performance improvements, and bug fixes. I’ve just tagged them both as 6.x-1.2-beta1 and hope that some people will help us with testing and the final polishing. There really are many improvements and new features, so I’ll just include notes and screenshots on some of them here. If you’d like to know about them all and have some more time, you can read the full change logs.

For the Messaging framework, perhaps the most interesting new feature is support for David’s Token Logic module (yeah, conditional tokens for templates!). The other changes – besides smoother integration with all available sending methods (SMS, XMPP, Twitter, Privatemsg, several mail engines…) – are focused on the administration UI. I hope it will now be much easier to set up.

Here are screenshots of the Messaging settings and the Messaging templates pages:

The Token module is implemented in many different Drupal contrib modules, where it provides simple run-time text replacement for predetermined messages. What this means is that an administrator or user can write something like this:

The brackets will then be automatically filled in with the current user's username. The Messaging and Notifications modules, maintained by Jose Reyero, use this functionality extensively when sending out messages. The site administrator configures a few generalized templates, which are then filled in correctly at run-time by Token for every message that is sent out:

However, this does have its limits - the biggest being that the text must be static. Notifications accepts different message templates for updates, creations, node events, etc., but this has limits in terms of implementation and usability. It just isn't practical to provide templates for every different permutation of node type, event type, and any other tokenized information that administrators can dream up.

But there are some obvious use cases. One of the most frequently requested features for Messaging and Notifications has been the ability to send out different message templates based on the content type of the node in question. An example of this is sending a notification like "[author-name] has just blogged about [title]" when a new blog post has been published, and a notification like "A new [type-name] has been posted" when anything else is published. Previously, the way to accomplish this was to code a custom token that read in the values and conditionally returned hardcoded text. This process was time-consuming and difficult to maintain at best, and at worst not even feasible, depending on an administrator's module-coding prowess.

Now with the new optional token_logic module, this functionality has been implemented - with no module coding required! Token logic adds five new tokens that can be input in the same manner as any preexisting tokens. These tokens allow for the basic branching in tokenized messaging based on the available token values. So, rather than being limited to just node type filtering, token_logic's tokens can be used to filter notifications based on anything from author name to term id to recipient name. Here's how this would look for the blog post use case:

We’ve done a lot to set up team communications tools that help us work more productively. So far we have an intranet with a HTML shoutbox and an XMPP server for secure instant messaging and group chats, and we use the Notifications and Messaging frameworks so we can subscribe and get notifications of updates from just about anywhere. Our next challenge is to put all these pieces together and have notifications and messages passed around from anyone to anywhere.

The missing piece to do this and glue these pieces together is a messaging broker that can take incoming messages from multiple input channels, do some processing with them, and route the results to anywhere. We recently got this working. Meet the Messaging processor, an architecture capable of handling multiple channels and configurable processing queues. This is a high level block diagram of how it all fits together.

Note: Though the following screenshots are from live sites, this is still in the development stage and we are not releasing all the code (yet). Stay tuned for upcoming releases of all these modules.

The Ingredients

• XMPP server and the XMPP Framework, with a few patches to allow persistent XMPP connections
• A (100% PHP Drupal) chatbot that can connect periodically to the chat room and post read messages
• A shoutbox module (integrated with our spaces architecture) that allows setting up shoutbox blocks for informal message blasts, tied to a specific team or site section
• Messaging and Notifications frameworks, which are fully released as contributed modules
• The new Messaging processor, which handles incoming messages, processing, and routing between channels

Keeping up with all the information we get every day is no easy job. We follow hundreds of websites using feed readers and get hundreds of emails every day. If that wasn't enough, there are new sites everyday producing their own streams of information, sites like Flickr, Facebook, Twitter, and probably someone is launching a new one as I'm writing this. We definitely need help to manage all this information, so we use tools like email filters, feed aggregators and readers, and tagging tools.

Just one of the tools mentioned above means one more piece of software or website to deal with. And that means that when we find some other tool that can mix what two tools do into just one, we have at least some hope that the next day we'll have some spare time to do the real work. That's what FeedAPI Mail does for FeedAPI in Drupal -  adds one more functionality to the mix. 

Now we have the first proof-of-concept module for a FeedAPI plugin that adds mailing lists into the equation. With this plugin, you can follow mailing lists and enjoy the same features available for the other content. It treats individual mailing lists as if they were web feeds, allowing the same features for incoming emails as for the rest of the content, including automatic tagging (and geotagging), reading the emails in a single news reader, easy sharing with your team, and so on. Here's a look at it running in our team news aggregator/analyzer Managing News:

This FeedAPI Mail module works together with Mailhandler and FeedAPI, which grabs the rest of the feeds and presents them in a single web interface. In addition it has also a specific mail reader UI that can display threads and authors, which puts single emails in context.

To get started we need a mail account and to set up a mail inbox with Mailhandler so emails are read and made into nodes. Next we need to add a 'Mailing list' and set it up with the mailing list email address so incoming emails can be classified as coming from different lists. Then we just need to subscribe our email address to the mailing list (which may take some manual steps to handle confirmation emails), and we're all set.

What's to Come with FeedAPI Mail

We're getting excited to come over for DrupalCon Szeged in just over a month. This will be the fifth DrupalCon that Development Seed has attended, and it always amazes me how much these conferences show off just how fast this community is growing and how far Drupal has come as a platform. We've looking forward to talking about some of our latest work to add and refine functionalities to Drupal. Here's a quick summary of the sessions we've proposed to lead. If you're interested in these topics, please vote for them!

Spaces and Context Modules: Tools for Site Building: The Context and Spaces modules are two relatively new tools in Drupal's aresenal that make it easier to build complex websites. In this session, Jeff Miccolis will talk about both modules' approach and show developers how they can be used and extended. He'll also show some examples of the modules in use on community portals, sites-within-sites, and intranets. Vote here.

Messaging and Notifications Frameworks: At DrupalCon Boston, Jose Reyero introduced the beta versions of these frameworks. A lot has been done to improve them since, and in this session Jose will talk about the upgrades, specifically in how they handle subscriptions, notifications of events, and the various delivery methods for sending messages. Another focus of this session will be to discuss the shift way from email only delivery methods to multi-platform methods. Vote here.

A New Aggregator for Drupal 7: Drupal's core aggregator is getting a revamp in Drupal 7. In this session, Aron Novak and Alex Barth will talk about why this step is needed and what you can expect from the new core aggregator. Aron began this work as a Google Summer of Code project last summer and this summer has continued to finetune it. The result is a simple yet extensible and efficient architecture that should serve Drupal well. Vote here.

Attracting and Retaining Drupal Talent: At the rate Drupal's popularity is growing, we're finding that there just aren't enough developers to meet the demand. Web shops and organizations are coming up short in finding the Drupal talent they need to build and run the online tools they want. This session will look at ways to beat out the hiring competition to find and retain Drupal talent. Eric Gundersen will talk about how Development Seed has grown our team, and Kris Krug from Raincity will share his experiences and lessons learned.

You can vote on all the sessions (including BoFs) you want to see at DrupalCon Szeged here. See you in Szeged!

The SMS Framework project has gathered a great amount of support from the developer community. Tylor at Raincity Studios has been doing some great work with fixing up the SMS Framework project, so I thought it would be good to return to my roots to upgrade the framework and contribute an important feature. We're moving toward a 1.0 release for 5.x and ultimately upgrading to Drupal 6, so stay tuned for that. 

As I've been moving down the issue queue reviewing and applying patches, one feature stood out to me - the need to store users' mobile numbers in a separate table, instead of in $user->data. This will allow Drupal to identify the author of incoming SMS messages. With this feature, your visitors will be able to get updates about various events that occur on your site. For example, they can be alerted when new comments are made on their posts or when a 'event' node type is created. The patch includes a restructured interface for adding and confirming your mobile number. It's very similar to Twitter and Brightkite :) The Messaging Framework makes use of the API to support SMS as a send method. Add Notifications and you'll be able to alert users about updated content via SMS. I wanted to quickly show everyone how easy this is to set up.

1. Download the SMS Framework, the Messaging Framework, and the Notifications module and place them in your modules directory. (Also get Token as Notifications depends on it.)

2. Turn on the following modules:

• Messaging Framework
• SMS Messaging
• Content Notifications
• Notifications
• Notifications UI
• Token
• Email Gateway (We'll be using this for the example, but any gateway module should work.)
• SMS Framework
• SMS User

3. Go to admin/smsframework/gateways and select and configure your default gateway

4. Go to admin/notifications/settings and check 'Notify poster of own posts' for testing purposes

5. Go to admin/notifications/settings/ui and enable the desired subscription types

6. Go to user/1/notifications/nodetype and subscribe to the desired node type

7. Go to user/1/edit or user/1/mobile (depending on your SMS Framework build) and follow the steps to set up and confirm your mobile number

8. Post a node of that type and wait for your SMS message!

The SMS Framework module is still under development and is not recommended for production use.

"Reply ABOVE this LINE to POST a COMMENT." This is how automatic email notifications from the teams' intranet blog posts look these days. And it's true! Now I can just reply to any email notification I receive and my comment will appear on our intranet. The only bad thing is that I don't have an excuse anymore for not replying right away ;-)

Typically there are two ways to carry on online discussions and to keep up with the replies - mailing lists (that may have a web interface) and web forums (that may have email subscriptions). Though each of one has its own advantages and drawbacks, trying to mix both usually ends up in long threads of differently formatted messages. Besides, mailing lists are usually exposed to spamming (worse yet if spam ends up on the web) and the "from" address is really easy to forge.

As usual, we wanted it all - the reliability and readability that come with authentication in a web forum and the ease of replying through a mailing list. Since we already had the Notifications and Messaging framework and an excellent mailhandler module to fetch incoming emails, it was just a question of putting the pieces together and adding some extra security/authentication into the recipe to get to this. 

Check the new "Mail2web" plug-in in the Notifications module. This first version is for Drupal 5. And by the way, we just released the Drupal 6 version of both Notifications and Messaging modules. The Mail2Web Drupal 6 version, however, will have to wait for the mailhandler module.

Being paranoid about security and spam, we added a more secure method for message authentication than the ones currently used. Our outgoing emails have some digitally assigned parameters that are checked when the reply comes back. Thus only emails posted as a direct 'Reply-To' operation will get in as responses. And replies to a given thread will get only into that thread, nowhere else. But best of all, these tokens are completely invisible for a regular user and should work with most email clients without the need to install any extra software. (You can set an expiration time frame for replies too).

So what do I need to to do to get it working?

Some time ago we released the Notifications and Messaging packages as two handy and flexible frameworks to handle user subscriptions, notifications, and multiple messaging methods.

The good welcome to these tools by other developers has pushed us to keep working hard on them and that means a lot of improvements have gotten into both frameworks since the initial release. Thanks to everyone who's been sending feedback and/or patches for them. We've also been using the frameworks for our Intranet for some time now (yes, they work!) so a lot of ideas and improvements have come from our day to day user experience.

Here are a few of the big improvements:

• Two new messaging plug-ins
• SMS developed by Tim Cullen
• PHPMailer contributed by Ted Serbinski
• Better message formatting, improved filtering, and nice HTML to text conversion
• Support for bulk messaging - not just for sending messages to single users anymore
• Many usability improvements and bug fixes

There are now lots of different plug-ins for different subscription types and more
coming. You'll be able to manage them all from your user account. 

Here's a look at the main messaging settings page. See the new SMS and HTML mail (Phpmailer) plug-ins enabled and available:

As a simple proof of how easy is to develop a new messaging method plug-in, both of the new ones are around 100 lines of code and that's counting comments and white lines :-). Also we're learning as we go and based on this experience,  plug-in support is now much improved so itwill be even easier to create new ones.

We'll be presenting this work at the Boston DrupalCon so if you are interested in better notifications for your website users, you shouldn't miss the Mail, Notifications, and Messaging Framework session next week.

Coming soon: Upgrading to Drupal 6

As we have these two powerful frameworks, FeedAPI and the new Notifications framework I talked about  in my last post, I was thinking it would be cool to have them work together. Users already had the option to stay subscribed (i.e. to all nodes created of a given type), but this didn't seem like a very efficient way of getting notifications for all updated feed items. 

So I’ve implemented this new notifications plug-in that allows you to subscribe to a feed and then get lists of new/updated items. To see it working please take a look at the notifications_feed.module in the notifications package.

The resulting email notification looks like this:

This simple module takes advantage of the existing Notifications and Messaging frameworks to implement - with just a few lines of code - a new event type (Feed update) and a new subscription type (Subscribe to feed) with all the features, such as configurable sending methods, queueing, and custom message templates.

The idea, in short, is that we can now easily create any custom plug-in to keep users posted about anything with just a few lines of code, while having all the gory details like message templating and delivery, user subscriptions,  notifications queueing for large sites, etc. automatically handled by these modules. 

And you don't need to create any new table or field for managing all this data!

The notifications_feed.module is already part of the Notifications Framework.

Do you want a nice Ajax UI that lets users subscribe to comments by a given user, but only for nodes of a given content type and that have a certain tag assigned, oh, and only for comments posted on Friday? Do you also want these notifications sent to you printed on nice postcards?

Then you should check out the Notifications Framework. Nope, it doesn't do all that - yet :-) - but you won't have to write it all from scratch. You'll only need to write the UI plug-in, a short module defining the parameters for this subscription type, and the 'print-and-send-to-the-post-office' module, which I think will be the most difficult one. Hey, and if you contribute back the 'print-and-send-to-the-post-office' module, I'll be able to use it with my 'happy-birthday' plugín to send birthday postcards to the users of my website!

Here's a look at the Notifications Framework in Drupal:

And here's how it looks in use on our intranet: