Tag: evaluation

Which self?

I recently watched this TED talk by Daniel Kahneman about the experiencing self and the remembering self.

Apparently, they’re quite different. The experiencing self is the one who lives and feels in the moment. The remembering self is the one that engages in retrospective sense-making and decides, post-facto, whether the experience was good, fun, etc. It is the remembering self’s evaluation that informs future decision making.

This has enormous implications for UX evaluation. Even if the experiencing self has a (relatively) bad time, as Kahneman explains in the talk, but the remembering self makes a positive evaluation, the experience is remembered as good. We can measure UX in the moment, and track eye gaze and all that jazz. But ultimately, what really matters for future decisions is what users take away from the experience and how they evaluate it after it’s over. This is good news. It means that users may forget or put up with a few frustrations – and still assess the experience well, especially if it ends well. It also means that the research framework for website experience analysis that I created back in 2004 is valuable, because it focuses on how users make sense of the experience and what they take away.

New Communications Measurement and Evaluation

[Notes from Track1-session1, New Comm Forum]

Blake Cahill, Visible Technologies overviewed a couple of case studies of Visible Technologies clients and their online conversation analysis efforts.

Janet Eden-Harris, Umbria. Umbria was created to tap into and aggregate “unstructured text” and mine conversations. Umbria analyzes patterns of conversation – different groups of people speak differently. Umbria focuses on 4 areas:

  1. Tell me about my brand – not by doing surveys, but by listening to spontaneous conversations taking place in social media
  2. Tell me about my industry – are there trends I need to be paying attention to?
  3. Tell me about my customers – many people blog, and they say a lot about things they care about. Blogs can help you understand what your customers care about. For example, Umbria analyzed blog posts by dog lovers. Interesting patterns emerged: Gen Y talk about pets as accessories “How do I look with my pet?” Gen X talks about how to integrate the pet in family life. Baby Boomers talk about pets as people/companions – they talk to them, dress them up, etc. 🙂
  4. Tell me about my product – online conversations produce ideas about new products. Many new products fail, but if you build a product based on existing conversations, it might be more likely to fill a need. Umbria grouped online conversations about pets into smaller topics. If the conversation is positive, this means that particular need is being met: e.g. organic food, pet daycare. Negative conversations revolved around traveling with pets. Umbria’s client, DelMonte, saw a product idea there. They created a line of products for traveling with pets and created online content and information about traveling with pets: tips, pet-friendly hotels, etc.

Q&A

Q: Is what you’re talking about monitoring or measurement?

A: If you track monitoring over time, it can become measurement. You don’t track (only) eyeballs anymore. You track the change of sentiment in online conversations, and ultimately, you need to see if the social media campaign ties back to sales.

Q: What is Umbria’s data universe?

A: Umbria has the tools to collect tens of thousands of blog conversations. You can never get them all, but for one client, you might analyze about 10,000 blog posts.

Q: Is traditional business segmentation falling apart? Can you trust computers to analyze the data and come up with categories?

A: You cannot trust computers. You have to oversee the data. But it’s not feasible for humans to analyze every single comment. Segmentation is not dead, but you have to understand that one person fits in different segments for different contexts. The same person might be a bargain-hunter when shopping for cleaning products, but think nothing of spending $5 for coffee at Starbucks. So segmentation makes sense in specific contexts. It can be a useful and powerful tool.

Q: How do you code data? Is “positive” the same for Dell, Ernst & Young and dog food?

A: Jane answers: At Umbria we use natural language processing algorithms to analyze comments and identify: age, gender, and sentiment. If you show comments to people and ask them to identify sentiment, inter-coder agreement will be only 65%. Algorithms can only get close to that, but they can’t get better than that. Sentiment is really hard to identify and code. So you have to keep working on teaching the software how to code and score. Also have to keep in mind that language and language patterns keep changing.

Q: What’s it going to take to move these technologies to analyzing more than text?

A: You don’t look at comments individually, you have to look at interaction and conversation threads. For audio and videos, we look to transcribing the audio to text and having it analyzed.

Q: Is there a way to assess if blogs are increasing or decreasing in importance?

A: K.D. Paine: Don’t ask me, ask your audience. There are all these tools out there, find out what your particular audience is using.

Q: If I’m a nonprofit and don’t have $10,000 to spend, what do I do?

A: Do a quick keyword search on technorati, etc. and look through the conversations. Even if you don’t do the detailed segmentation we do, you can get a very good idea of online conversations on topics you care about. If you don’t use any tool or technology, you just have to read the posts & comments. So, what do you look for when you read them? You can note sentiment, visibility, themes & trends that emerge from those conversations, etc. Richard (Dell) explains you shouldn’t be afraid of going through comments manually. If you have the right searches set up, it takes 60-90 minutes to go through about 1,000 post. Richard responds to about 15% of blog posts. It’s likely that a nonprofit doesn’t have the same volume of comments as Dell, so reading comments using a feed reader is entirely feasible.

Q: Have you thought about open-sourcing your algorithms so smaller companies can use them?

A: Blake: We’re still working on improving those algorithms. We’ve been focusing on perfecting our technology. We’re operating at the enterprise level, but there are many people out there who provide basic services at very low prices. Jane: Companies will probably not do this, but there are tools being developed in academia which will become public domain.