Development Seed Blog

What's your approach?

This Wednesday we finished a migration from an old site running on OpenACS to a relatively complex Drupal site with organic groups, events, imagecache, attachments, books, and the whole spiel.

Having gone over budget on migrations before, this time we wanted to take a more systematic approach and make sure that we at least learned something if we wind up eating hours.

This is the approach we came up with:

1) Analyze, analyze, analyze

2) Write reusable code :)

3) Share our experience

1) Analyze, analyze, analyze

A problem in several previous migrations was that the real time hogs appeared late on the scene. Problematic data webs passed under our radar when estimating projects but took all our attention when the days got late. Another issue we had run into was that seemingly simple tasks turned out to be quite twisted and time consuming.

For this migration we did an exhaustive analysis of the legacy system and, most importantly, we did a detailed breakdown of tasks with hour estimates next to them. This allows us now to compare estimates with the actual work and draw conclusions that will improve our future estimates.    

2) Write reusable code

During past projects, we aggregated a good stock of code pieces for migrations, but they weren't reusable out-of-the-box. For every new migration project, functions needed to be copied and reconfigured. This meant that every time we created a couple new bugs during the reconfiguration and lost some of the hard earned experience with the quirks of programmatically creating content in Drupal.

With this migration, as much as possible, we wanted to  write project independent, reusable functionality that would cast solved problems into code for future migrations.

We whipped up a migration framework that does four things:

• defines a simple call back based API for implementing chunks of migrations
• handles backups and roll backs of migrations 
• provides an organized place for collecting reusable create and helper functions
• offers a legacy -> new id mapper 

We call this framework MTK - Migration Toolkit - and here is a screenshot of it: 

It's somewhat similar to Migrator module. The reason we went with our own concoction is simply because we had so much to learn and discover as we went. If you're interested, check out the code for MTK in my sandbox.

3) Share your experience

Well, this is what I'm doing right here :)  Have you been bitten by the migration time suck bug before? Have you found a remedy against it? I'm curious to know what your tricks to avoid it are. 

Now as we've put this migration behind ourselves, I'd say that the above approach made the task less daunting and more rewarding since we didn't just kill a nasty beast but also improved our toolkit in a sustainable way. Of course we had some hard nuts to crack, but as a result we've got pretty decent nut crackers. We learned.

One thing we couldn't get past though was digging through the legacy database and doing the popular what-is-what game. In our office, Tim Cullen has coined the telling term 'data archeology' for this. Data archeology will stay with us for a while. The legacy site we migrated from was built in 2004. I wonder if the person that migrates from the 'new' site in 2012 will have a much better handle on migration than we have right now.