New Communications Measurement and Evaluation

[Notes from Track1-session1, New Comm Forum]

Blake Cahill, Visible Technologies overviewed a couple of case studies of Visible Technologies clients and their online conversation analysis efforts.

Janet Eden-Harris, Umbria. Umbria was created to tap into and aggregate “unstructured text” and mine conversations. Umbria analyzes patterns of conversation – different groups of people speak differently. Umbria focuses on 4 areas:

  1. Tell me about my brand – not by doing surveys, but by listening to spontaneous conversations taking place in social media
  2. Tell me about my industry – are there trends I need to be paying attention to?
  3. Tell me about my customers – many people blog, and they say a lot about things they care about. Blogs can help you understand what your customers care about. For example, Umbria analyzed blog posts by dog lovers. Interesting patterns emerged: Gen Y talk about pets as accessories “How do I look with my pet?” Gen X talks about how to integrate the pet in family life. Baby Boomers talk about pets as people/companions – they talk to them, dress them up, etc. 🙂
  4. Tell me about my product – online conversations produce ideas about new products. Many new products fail, but if you build a product based on existing conversations, it might be more likely to fill a need. Umbria grouped online conversations about pets into smaller topics. If the conversation is positive, this means that particular need is being met: e.g. organic food, pet daycare. Negative conversations revolved around traveling with pets. Umbria’s client, DelMonte, saw a product idea there. They created a line of products for traveling with pets and created online content and information about traveling with pets: tips, pet-friendly hotels, etc.


Q: Is what you’re talking about monitoring or measurement?

A: If you track monitoring over time, it can become measurement. You don’t track (only) eyeballs anymore. You track the change of sentiment in online conversations, and ultimately, you need to see if the social media campaign ties back to sales.

Q: What is Umbria’s data universe?

A: Umbria has the tools to collect tens of thousands of blog conversations. You can never get them all, but for one client, you might analyze about 10,000 blog posts.

Q: Is traditional business segmentation falling apart? Can you trust computers to analyze the data and come up with categories?

A: You cannot trust computers. You have to oversee the data. But it’s not feasible for humans to analyze every single comment. Segmentation is not dead, but you have to understand that one person fits in different segments for different contexts. The same person might be a bargain-hunter when shopping for cleaning products, but think nothing of spending $5 for coffee at Starbucks. So segmentation makes sense in specific contexts. It can be a useful and powerful tool.

Q: How do you code data? Is “positive” the same for Dell, Ernst & Young and dog food?

A: Jane answers: At Umbria we use natural language processing algorithms to analyze comments and identify: age, gender, and sentiment. If you show comments to people and ask them to identify sentiment, inter-coder agreement will be only 65%. Algorithms can only get close to that, but they can’t get better than that. Sentiment is really hard to identify and code. So you have to keep working on teaching the software how to code and score. Also have to keep in mind that language and language patterns keep changing.

Q: What’s it going to take to move these technologies to analyzing more than text?

A: You don’t look at comments individually, you have to look at interaction and conversation threads. For audio and videos, we look to transcribing the audio to text and having it analyzed.

Q: Is there a way to assess if blogs are increasing or decreasing in importance?

A: K.D. Paine: Don’t ask me, ask your audience. There are all these tools out there, find out what your particular audience is using.

Q: If I’m a nonprofit and don’t have $10,000 to spend, what do I do?

A: Do a quick keyword search on technorati, etc. and look through the conversations. Even if you don’t do the detailed segmentation we do, you can get a very good idea of online conversations on topics you care about. If you don’t use any tool or technology, you just have to read the posts & comments. So, what do you look for when you read them? You can note sentiment, visibility, themes & trends that emerge from those conversations, etc. Richard (Dell) explains you shouldn’t be afraid of going through comments manually. If you have the right searches set up, it takes 60-90 minutes to go through about 1,000 post. Richard responds to about 15% of blog posts. It’s likely that a nonprofit doesn’t have the same volume of comments as Dell, so reading comments using a feed reader is entirely feasible.

Q: Have you thought about open-sourcing your algorithms so smaller companies can use them?

A: Blake: We’re still working on improving those algorithms. We’ve been focusing on perfecting our technology. We’re operating at the enterprise level, but there are many people out there who provide basic services at very low prices. Jane: Companies will probably not do this, but there are tools being developed in academia which will become public domain.