Development Seed Blog

Hot Tags and the Tags Related to Them

Think about the news you track and the issues you need to stay on top of on a daily basis. For me, one of those topics is social networking. There are often thousands of stories about social networking published every day with perhaps hundreds of different angles to them. Maybe somewhere in those thousands of stories and hundreds of angles is an emerging trend – major research on its impact on job hunting, a new technology or service that the blogosphere is drumming up, or the use of these networks by a new segment of the population. But how can I spot the trends in all that news?

Well, graphing the news can help. When you look at the incidence of a word in the news and can see the rise and fall of its use on a day-to-day basis, you can tell if a subject is gaining momentum in the media. However, seeing the rise and fall of a word’s incidence does nothing to help determine why it’s getting more or less attention. To figure that out you need more information. One way to do this is to look at words related to the hot topics in the media.

This graph shows the mentions of MySpace in the media from July 8 to July 30 (data from Trendio.com*). You can see that there is a slight spike toward the end of the chart and that mentions for MySpace peak at 164 on July 27. But why? 

We working on a tool – a fully loaded team aggregator called Managing News - that can help you figure out in just a few seconds why spikes like this one are happening.  Managing News tracks the daily mentions of words – called tags – in the media and plots the topics most in flux on graphs. Now, of course you're not able to read all these headlines, never mind the full articles, but you can quickly see the spikes in prevalence of a word. But you still need to figure out why these spikes are happening. What if you could see other keywords mentioned in the same articles? A quick glance at the related tags can help filter up related and even causal words, and it will give you a good idea as to why a topic is getting media attention. 

This is where a service like Yahoo Terms Extraction and good statistical analysis come in to help you cut through the information overload. Managing News runs Yahoo's Terms Extraction service across all articles that come into the system and assigns the articles tags and keywords that show the main context of the articles. Then it relates these terms to each other. 

Going back to the MySpace example, mentions of the social networking platform spiked and then peaked on July 27. The most frequently used words related to articles mentioning MySpace were "sex offenders," "attorney general," "Roy Cooper," "legislation," and head of MySpace security "Hemanshu Nigam.” And what happened on July 27? The company announced the detection and removal of 29,000 registered sex offenders from their popular social networking platform.

Associating related tags to hot topics is also helpful for topics that are consistently in the news, like Iraq. Iraq might be a keyword in 2,000 articles from the sources you track every day, and these articles may focus on 75 different topics. By looking at related words, you can quickly cut through the weeds and figure out what is hot in Iraq news today.

This and other features will soon be highlighted on a new Managing News website due out soon to share details about the tool. Better keyword extraction and great presentation of data on hot tags and their related tags are just some of the many improvements slated for the new Managing News.

* Trendio is a "current events stock exchange" where the "words from the news are the stocks" in a community-based game.