When you’re keeping tabs on an event like the upcoming election in Afghanistan, a basic street map that plots news stories is quite useful. But what could you do with a map that plots those news stories over voting regions that are shaded by poverty rate, literacy rate, or another human development indicator? The effectiveness of a map increases drastically when you add specialized data to the base layer. In this case, not only would you see the hot spots of activity, you could identify possible explanations for the activity.

The maps we’re familiar with are powered by tile sets – collections containing hundreds of thousands of individually rendered images that stitch together to form a larger map view. Tile sets are useful because they allow users to pan and zoom around a map with a web browser, but creating and maintaining a tile set is challenging. Tile generation demands a considerable amount of computing power and can take days depending on the size of the region being rendered. Finished tile sets occupy many gigabytes of disk space, making storage and distribution difficult.

With the help of Amazon Web Services, we’re building an infrastructure capable of generating beautiful interactive maps quickly. We’re using four Amazon services in this workflow: SQS (job queuing), EC2 (tile generation), S3 (storage), and CloudFront (distribution). The figure below illustrates the design.

This week is a short one for most with Independence Day giving many of us in the United States a three day weekend. Typically for the 4th of July in Washington, DC you see a lot of people leave town to celebrate, while the tourists flood in to do the same. But if you are in town this week and weekend, there are some fun things happening with Artomatic celebrating its last week of exhibits and shows, the Smithsonian Folklife Festival going on for a second week on the Mall, and of course the fireworks on the 4th.

In addition to being a short week, it’s also a quiet one in technology. There are just a couple events on our radar this week, which you can see below. For a full list, check out DC Tech Events. Have a great week, and enjoy the holiday weekend!

Tuesday, June 30

7:00 pm

Online News Association Meetup: This month’s meetup for online media folks will feature a panel on government 2.0 where people working in this area will discuss what’s being proposed and acted on, and will showcase some of the apps, ideas, and technology being developed to improve government transparency and collaboration.

Wednesday, July 1

6:30 – 9:30 pm

Northern VA Drupal Meetup: If you’re a Drupal developer, wannabe Drupal developer, or just want to learn more, check out this meetup. It’s a great chance to get to know people working with Drupal, see what’s possible with the platform, and get answers to your questions. The floor will be open for short presentations, so consider talking about the latest Drupal project you’re working on.

Recently FeedAPI 1.7beta2 was released, and this new release has Drush support. Drush, in a nutshell, lets you handle your Drupal instance with standard unix command line tools (DRUpal SHell), and with Drush support now in FeedAPI, you can better control your feeds. In this post, I'll introduce you to a way to refresh your feeds faster and more efficiently.

This method is only useful when your Drupal site runs on a VPS or a dedicated box, otherwise you cannot really use Drush at all. The script uses multiprocessing and starts refreshing four feeds at once. However, it's not a true parallel since the next four feeds will only start refreshing after all of the four previous feeds are finished. You can also adjust the number of feeds that you'd like to start in a round. The script is basically an infinite loop, but don't worry, it won't eat up all your server resources. If all of your feeds can be refreshed in a short period of time (900 seconds, configurable), the script will wait a bit before it starts from the beginning again.

Imagine a similar solution in pure PHP. It would have to be really hackish because of the lack of parelell/multithreaded constructs in PHP. And needless to say, with this script you don't have to worry about memory limits and timeouts (you can have a separate php conf for the command line environment).

Here is the script:

Yesterday, Eric talked about how feature servers can help organizations manage and update features on different of sites and I pulled together a screencast of an initial proof-of-concept. While we’ve been talking about these ideas in the office for a while, the actual implementation is very new and I was surprised at how quickly it went – in all, the feature server took less than a day to build.

Here’s a look at the feature server with a prepackaged theme (more on that below):

What it does

• Lets you create projects and releases and manage them by hand on a Drupal site.
• Produces an update XML feed compatible with the update module in core.

What it doesn’t do

• Version control integration and automated packaging. The project module does an awesome job of this, and if you’re serious about tying your features into VC it’s what you’ll want to look at.
• Anything with dependency / build management systems (a problem that Adrian Rossouw and others are spearheading in the community).
• Info file based parsing, dependency detection, and display.

The ingredients

Adrian pointed out to me that this very simple version of a feature server could be a feature itself. It turns out he was right – it is largely generic site building with some additional custom components:

Today a group of 50+ experts in public health, communications, and defense from both the private and public sectors are gathering at the National Institute of Health to talk about pandemic preparedness and community resilience around the H1N1 flu. So far the talks have been great, with Vint Cerf and David Alberts leading off this morning, Chris Allen talking about decentralized open source communication tools, and Larry Brilliant scheduled to talk specifically about H1N1 this afternoon.

In addition to talks by big thinkers like these, there are also workings sessions that will focus on addressing “mission critical gaps” around the possibility of a second wave of an H1N1 pandemic hitting this fall. Eric will be presenting in one of these on the decentralized communications tools we are working on and how they can help in a pandemic – specifically looking at Open Atrium and Managing News. He’ll also be talking about how distribution tools like Drupal’s new feature server can allow for more resilient software development communities.

You can listen to the event here where they’re live streaming it. Tune in at 3:00 pm to hear the working groups report back on what they’ve discussed (these won’t each be streamed).

It's common for organizations to run 20+ different program websites. The bulk of these program sites need several of the same features like a public blog, a pressroom, and media resources. A few key programs might need a sensitive CCK-based reporting feature that was designed in house.

This screencast by Young, who blogged about the power of decentralized features yesterday, is a first look at how a feature server will let you pull features from multiple sources and get updates for them in both public and potentially secure and private ways. If you're not familiar with the features module, you may want to watch the screencast introducing it before watching this one.

The feature server architecture is incredibly simple because it leverages core Drupal components to their fullest. Believe it or not, the feature server is itself a feature (exportables for node types, CCK fields, Views, etc.) with a few small custom components (e.g. a Views style plugin for generating the update module-compliant XML).

After my previous post introducing Features, people had some very good questions. How can you share features that you’ve built? And how can other users get your features?

We think the answer to these questions is to use a feature server. At its simplest, a feature server is a website where someone can get features and get updates for those features. It could be a private repository for a network of Drupal sites within an organization, or it could be a public facing site from which anyone can download and add features to their own Drupal website.

Stumble Safely got a great mention in the July issue of Wired : ) In an article on Vivek Kundra, the CIO of the United States, and his missive to get government data online, in one place, and open and accessible (aka Data.gov), Stumble Safely was mentioned as an example of what can come from opening up data like this. We built the site in last year’s Apps for Democracy contest, which was an effort Kundra backed while CTO of Washington, DC to get people to build apps that use DC’s government data.

Here’s what was said in Wired:

In DC, someone combined several of the data sets released by local government – maps, liquor license info, crime statistics – into an app called Stumble Safely, which shows users the safest way to walk home when drunk.

Yep, we’re the “someone” who built Stumble Safely and you can guess where our need for this app came from : )

It’s an honor to be included as an example of the positive results of opening up government data in Wired. Thanks Vivek and Nicholas Thompson for the shout out! You can read the full article here.

It’s officially summer! But that doesn’t mean that the city slows down – well, at least until August. This week there are quite a few technology events happening in Washington, DC. Below are a few that caught our eye. You can find a full listing of the week’s tech events here.

Monday, June 22

6:00 – 8:00 pm

Mobile Monday DC: This monthly meetup for people using cell phones for business and good will look at what the recession means for mobile advertising – and how it’s affecting the organizations and companies using it for fundraising and sales.

Tuesday, June 23

12:15 – 1:45 pm

Media Future Now: This brown bag lunch will feature a conversation on ways newsmakers are using social media from the viewpoint of two editors with different audiences – the Washington Post’s NFL editor and the founder of the Social Times, an online publication that covers what’s covered in social media.

6:00 – 8:00 pm

DC PHP Beverage Subgroup: At this monthly meetup, php coders get together to talk code and drink beer. If you like to do both of these things, you should probably check this one out.

How fast can you install modules? Drush, the Drupal command line utility, takes what normally is a tedious task and turns it into a delightful experience.

Final Times
Manual Method: 2:38:75
Drush Method: 1:17:57

Module installation is just one of the things Drush can help you with. Learn what else it can do for you on the project page: http://drupal.org/project/drush.