New Communications Measurement and Evaluation

[Notes from Track1-session1, New Comm Forum]

Blake Cahill, Visible Technologies overviewed a couple of case studies of Visible Technologies clients and their online conversation analysis efforts.

Janet Eden-Harris, Umbria. Umbria was created to tap into and aggregate “unstructured text” and mine conversations. Umbria analyzes patterns of conversation – different groups of people speak differently. Umbria focuses on 4 areas:

  1. Tell me about my brand – not by doing surveys, but by listening to spontaneous conversations taking place in social media
  2. Tell me about my industry – are there trends I need to be paying attention to?
  3. Tell me about my customers – many people blog, and they say a lot about things they care about. Blogs can help you understand what your customers care about. For example, Umbria analyzed blog posts by dog lovers. Interesting patterns emerged: Gen Y talk about pets as accessories “How do I look with my pet?” Gen X talks about how to integrate the pet in family life. Baby Boomers talk about pets as people/companions – they talk to them, dress them up, etc. 🙂
  4. Tell me about my product – online conversations produce ideas about new products. Many new products fail, but if you build a product based on existing conversations, it might be more likely to fill a need. Umbria grouped online conversations about pets into smaller topics. If the conversation is positive, this means that particular need is being met: e.g. organic food, pet daycare. Negative conversations revolved around traveling with pets. Umbria’s client, DelMonte, saw a product idea there. They created a line of products for traveling with pets and created online content and information about traveling with pets: tips, pet-friendly hotels, etc.


Q: Is what you’re talking about monitoring or measurement?

A: If you track monitoring over time, it can become measurement. You don’t track (only) eyeballs anymore. You track the change of sentiment in online conversations, and ultimately, you need to see if the social media campaign ties back to sales.

Q: What is Umbria’s data universe?

A: Umbria has the tools to collect tens of thousands of blog conversations. You can never get them all, but for one client, you might analyze about 10,000 blog posts.

Q: Is traditional business segmentation falling apart? Can you trust computers to analyze the data and come up with categories?

A: You cannot trust computers. You have to oversee the data. But it’s not feasible for humans to analyze every single comment. Segmentation is not dead, but you have to understand that one person fits in different segments for different contexts. The same person might be a bargain-hunter when shopping for cleaning products, but think nothing of spending $5 for coffee at Starbucks. So segmentation makes sense in specific contexts. It can be a useful and powerful tool.

Q: How do you code data? Is “positive” the same for Dell, Ernst & Young and dog food?

A: Jane answers: At Umbria we use natural language processing algorithms to analyze comments and identify: age, gender, and sentiment. If you show comments to people and ask them to identify sentiment, inter-coder agreement will be only 65%. Algorithms can only get close to that, but they can’t get better than that. Sentiment is really hard to identify and code. So you have to keep working on teaching the software how to code and score. Also have to keep in mind that language and language patterns keep changing.

Q: What’s it going to take to move these technologies to analyzing more than text?

A: You don’t look at comments individually, you have to look at interaction and conversation threads. For audio and videos, we look to transcribing the audio to text and having it analyzed.

Q: Is there a way to assess if blogs are increasing or decreasing in importance?

A: K.D. Paine: Don’t ask me, ask your audience. There are all these tools out there, find out what your particular audience is using.

Q: If I’m a nonprofit and don’t have $10,000 to spend, what do I do?

A: Do a quick keyword search on technorati, etc. and look through the conversations. Even if you don’t do the detailed segmentation we do, you can get a very good idea of online conversations on topics you care about. If you don’t use any tool or technology, you just have to read the posts & comments. So, what do you look for when you read them? You can note sentiment, visibility, themes & trends that emerge from those conversations, etc. Richard (Dell) explains you shouldn’t be afraid of going through comments manually. If you have the right searches set up, it takes 60-90 minutes to go through about 1,000 post. Richard responds to about 15% of blog posts. It’s likely that a nonprofit doesn’t have the same volume of comments as Dell, so reading comments using a feed reader is entirely feasible.

Q: Have you thought about open-sourcing your algorithms so smaller companies can use them?

A: Blake: We’re still working on improving those algorithms. We’ve been focusing on perfecting our technology. We’re operating at the enterprise level, but there are many people out there who provide basic services at very low prices. Jane: Companies will probably not do this, but there are tools being developed in academia which will become public domain.

SNCR New Communications Forum opening keynote: Joseph Jaffe

[cross-posted from New Communications Review, I’m live-blogging the 2008 SNCR New Communications Forum]

Notes from Joseph Jaffe’s opening keynote:

There are millions – millions of conversations going on around us: powerful, engaged, influential conversations: Isn’t it time we join them?

The world has changed. The consumer has changed. Why hasn’t marketing?

Is the consumer in control? Are organizations in control? Nobody’s in control – this is total anarchy. We seek order and control, but that’s a fantasy world, a false sense of control. Some organizations, such as P&G, are trying to give that control to consumers – but consumers don’t want control. They want to be listened to, respected, engaged.

Marketing theory is very outdated – based on simple top-of-mind associations. Would you rather have 10 quality relationships or 5 million impressions? Most marketers want the impressions. See the Comcast example – the first thing that comes to mind (and in search rankings) is the video with the Comcast technician falling asleep. That gets a lot of impressions, but are they valuable? If you cultivate 10 quality relationships, those will branch out and in the long term will bring much more value than impressions.

Moving marketing from 4 P’s to 6 C’s: Content, Customization, Commerce, Conversation, Community, Content – all revolving around the Consumer.

The history and future of media: Moving from the one-to-many model to the one-to-one model (personalization), to one-from-me (search). But the model that characterizes social media is the many-to-many model. The model can work, or can suffocate you if you don’t know how to listen. In some ways, social media has gone to a reverse one-to-many model, where the previous targets (consumers) have become the broadcasters.

Communication vs. Conversation: Communication can only take you so far. Communication gets your foot in the door, but conversation gets the consumer to open the door and invite you in. The way you do it is not to spend lots of money: “Don’t outspend… outsmart.”

SNCR survey: What does conversational marketing mean to you? Some definitions include: less hype, faith & trust in your brand, partnering with consumers, etc. Going back to the Cluetrain Manifesto and the ancient bazaars, those people knew conversations: those markets were conversations. Commerce was a social experience. But we’ve moved from the bazaar/market model to impersonal malls. It’s time to put the social & sociability back in media – and in marketing. Marketing doesn’t have to remain a spectator sport. It’s time to entice consumers to come out of spectator/lurker mode.

In a recent SNCR survey, 41% of respondents anticipated spending 10% or more of marketing budget on conversational marketing. Many CMO’s right now get it, and are ready for conversational marketing. You want your customers to trust you – but do you trust them?

The answer is not to spend absurd amounts of money on fireworks displays (advertising) – how many Superbowl ads can you remember?

It’s time to blend and mesh the worlds of marketing, PR, and advertising, to achieve transformational change. The biggest risk is to spend $4 million on a campaign no one notices. 90% of advertising is wasted. It’s time to reconsider our strategy.

The problem is we don’t nurture the young, fresh smart ideas and we don’t invest enough time in them. The seeds of conversation are not magic beans. Conversation will not happen overnight. Companies need to invest long-term in conversation and maintain that commitment.

At the end of the day, I’m a storm chaser. If you want to understand change, you have to be in the heart of the storm. I stick my neck out. It’s been chopped off a couple of times – but it keeps growing back.

Predictions based on recent SNCR research data:

  • by 2012, companies will have conversation departments. They will listen to customers. They will allow employees to engage with customers. They will sponsor the sand box – and create spaces for customers to engage with each other
  • campaigns will change radically to include influence outreach, blogger activation, conversation monitoring & optimization, etc.

So, where’s the catch?

  • talent: not enough people in the business right now. Advertisers are part of the problem. They’re not the creators; customers are.
  • measurement: not ROI, but return on experimentation; or return on infinity. Right now, there are 4 categories: sales, good PR, improved brand health, more intimate knowledge of customers – the last 2 are the most interesting.
  • integration: 55% of survey respondents stated that conversation should be a combination of marketing, PR, & advertising.
  • organizational buy-in: right now, change happens at the individual level, not at the organizational culture level; but that often means that change happens at the tactical level, and it does not radically change the way the organization operates
  • control: you lose it. Recall T-mobile claiming they own trademark on the color magenta and issuing a cease & desist letter to engadget. In response, many bloggers displayed “T-mobile sucks” magenta badges. 3H: humanity, humility, & humor.

Interview questions

On the PR network PROpenMic (if you haven’t joined yet, I think you should!), Phil Gomes, VP Edelman Digital answers questions from PR students & faculty. Below is the most recent video, in which Phil and a series of other Edelman employees answer the question:

What questions do you always ask in a job interview? Find more videos like this on PROpenMicIf you are a PR practitioner reading this, please add your favorite interview questions in the comments. If you’re a student, feel free to add an interesting/difficult interview question you have been asked in the past.


Every day, I feel bad about not posting on this blog. If I could get two thoughts together, I would.

Those who follow me on twitter know that I traveled to Romania over Spring break to visit my grandmother, who’s going through a very slow and painful recovery after hip replacement surgery.

Fortunately, this is the first time in my life so far that I’ve had to deal with a loved one’s suffering. I’m not dealing with it very well. It’s consuming my peace of mind, affecting my clarity of thought, and pretty much pushing me towards depression. I’m not proud of not being mentally strong enough to deal with this better… and thus, more guilt and more distress.

So, yes, her ups and downs have been affecting my life & my work. When she has a good day, I’m full of energy and ready to take on the world. Then she has a relapse (like today, when she did not get out of bed even once because of debilitating back pain) and my mind becomes a messy foggy worried knot.

I’m trying to learn about equanimity and offering compassion without letting my mind be deeply troubled – a Buddhist concept that takes years of practice to achieve.

Until I work through these things and figure out a way to bring my mind back to work, and social media, and things that all of a sudden seem dangerously unimportant, I am retreating into silence.

If I have something good to offer you, I will. Like this: PR Open Mic, a shiny brand new and very promising social network for PR people (students, faculty, and practitioners). I hope you join.

If you have any advice for me, any tips & tricks for dealing with this situation compassionately but with only a healthy amount of empathy, please let me know.