Tag: research methods

Website Experience Analysis

This post explains an alternative research protocol, website experience analysis (WEA).

Website experience analysis is a research protocol (set of procedures) that can help researchers identify what specific interface elements users associate with particular interpretations.

WEA focuses on the messages that users take-away from their experience with the interface.

All interfaces try to communicate something, such as:

  • you should trust this application with your credit card data
  • you should come study for a MS degree in CGT at Purdue
  • etc.

WEA allows you to find out:

  1. whether the interface actually communicates this message – do people actually take away the message that you intended, and to what extent?
  2. what specific elements of the interface users associate with those particular messages (trust, CGT is a good program, etc.)

The WEA questionnaire is based on prominence-interpretation theory. It works with pairs of items that ask:

  1. Ratings of user perceptions (e.g. trust – on a scale of 1-10)
  2. Open-ended: what about the interface makes the user feel this way?

WEA is based on a much more complex theoretical framework of the website experience. The framework breaks the website experience down into two major dimensions: time and space. WEA then explains the phases of the experience as they unfold across time, and the elements of the website space (elements are categorized according to element functions). The theoretical framework is likely only valid for websites, because the experience with another type of interface, even though it may have the same three main temporal phases (first impression, engagement, exit) will likely differ in terms of the steps within those phases and the nature of the spatial elements and their functions.

WEA is different from a regular questionnaire because it connects perceptions with specific interface elements. Questionnaires will tell you whether the user trusts the product, but they won’t provide specific feedback as to what particular elements may account for that perception.

WEA is modular, which means that a different battery of items can be used, depending on the focus of the research. I used WEA in 2 contexts:

  1. To evaluate the experience of visiting organizational websites. Here, I used the 5 dimensions of good relationships between organizations and their publics: trust, commitment, investment, dialog, etc.
  2. To evaluate whether emergency preparedness websites persuade users to take emergency preparedness actions. Here I used a battery of items derived from a theory of fear appeals (EPPM) and assessed whether users perceived there is a threat, believe they can do something about it, believe the recommended actions would be effective, etc.

I think WEA would provide excellent feedback about how prospective students perceive the CGT department, based on their experience with the website. It would be very valuable to find out exactly what about the website makes them feel that:

  • they would benefit from a CGT MS
  • they would fit in
  • they would have a good educational experience
  • etc. – we have to determine the relevant set of items. Ideally, we would have a theory to guide item development.

WEA can be used with other research questions, such as: How do HR managers look at job candidates’ online information? (hello, Jack!)

WEA can be improved upon to better tap into emotional aspects of the user experience. It can be modified to be a more inductive approach, that elicits emotions and interpretations from users rather than asking about specific interpretations (such as trust, etc.)  – thank you, Emma, for these suggestions!

If you would like to read more about WEA, you can find the relevant citations in Google Scholar. I can provide copies of the papers if you don’t have access to them.

Research Study: @sockington is more influential than @chrisbrogan

This Webecology research report has been making the rounds on Twitter. I haven’t had time to read it until now, here are my reading notes:

The Webecology team uses large scale data mining to identify patterns indicative of online culture and community. Wish I’d do this, too – and will, as soon as I find a research partner to help with the data mining part.

For this project, the authors set out to create a more accurate measure of influence on Twitter that goes beyond either:

  1. number of followers; or
  2. followers/friends ratio

The authors defined influence on Twitter as:

influence on Twitter = the potential of an action of a user to initiate a further action by another user

Specifically, influence means the potential of a tweet to generate replies, mentions (conversational behaviors), RTs, and attributions (content-pushing behaviors).

This is an atheoretical, operational definition of influence (the study’s Achille’s heel).

As far as I understand, all 4 actions were weighed equally. So, a RT factors the same as an @reply in determining influence.

They selected 12 Twitter accounts to study. The selection was based on this criterion: the 12 accounts were  “widely perceived to be among the more influential users on Twitter.” It is not clear who did the perceiving, and what definition or measure of influence they used in the process of perception. IMO, the arbitrary selection of the sample is another major weakness – but in this case, I can live with it, because the purpose is not to derive conclusions about Twitter culture as much as it is to demonstrate how the methodology can be used.

Then, the 12 users were grouped into 3 categories. Here is a table with the accounts they analyzed, and their number of tweets over 10 days, as well as the number of followers and friends at the end of the 10 days:

—

Celebrities Username Tweets Followers Followees
Ashton Kutcher aplusk 3,205 3,407,385 209
Shaquille O’Neil THE_REAL_SHAQ 2,072 2,092,541 562
Stanley Kirk Burrell MCHammer 6,016 1,331,797 31,202
Sockington sockington 5,711 1,089,984 380
Justine Ezarik ijustine 7,718 605,441 3,039
News Outlets Username Tweets Followers Followees
CNN Breaking News cnnbrk 1,096 2,712,530 18
BarackObama.com BarackObama 330 2,018,016 761,851
Mashable.com mashable 17,914 1,363,510 1,925
CNN cnn 11,607 193,625 50
Social Media Analysts Username Tweets Followers Followees
Gary Vaynerchuk garyvee 7,532 862,790 9,683
Chris Brogan chrisbrogan 48,341 94,715 88,431
Robert Scoble Scobleizer 23,112 94,295 2,423

The data that they mined was as collected over 10 days, in August 2009. The data included:

  • The 2143 tweets generated by the 12 users
  • The 90,130 actions (responses, RTs) triggered by the original 2143 tweets
  • All the tweets generated in connection with the 12 users (by their followers and friends;a total of 134, 654 tweets, 15,866,629 followers, and 899,773 friends/followees)

The authors produced 2 types of influence reports, based on the type of action that was triggered:

  1. conversational action (people replied, or mentioned the user – e.g. “meeting @stockington for catnip”)
  2. content-pushing action (people retweeted, or gave attribution – e.g. “via@username”)

Please note that a mention may or may not be a response to a tweet. If they were not responses to a tweet, they fall outside the authors’ definition of Twitter influence, and they should have been excluded from the analysis.

Here we go, on to the findings:

Conversational action

This graph shows you the amount of conversational activity (@replies and mentions) each user got in response to one (average) tweet.

Content action

This graph shows you how much content action (retweets and attributions) each user got for each (average) tweet:

So here we see that, per tweet, @sockington did get more retweets than @chrisbrogan.

The authors claim that these graphs of influence/tweet are the most accurate measure of Twitter influence so far. Therefore:

@sockington IS more influential on Twitter than @chrisbrogan,

because the fake cat gets more retweets. (sorry, @sockington, I do love you!!!)

I know exactly what you’re thinking, it starts with B and ends with T.

That’s because here we have a problem of construct validity. The measures do not actually measure influence. I wish the authors had read some research in communication & persuasion about the concept of influence, then worked their way from a conceptual to an operational definition.

Obviously, @sockington gets more retweets because he’s cuter & funnier than @chrisbrogan (sorry, Chris!). We don’t know why people reply or retweet. This study ignores a very important aspect of human relations: meaning. There is meaning in tweets, and meaning in why people retweet. But that is not captured in this study.

That being said, the report shows what can be done with data mining – it’s awesome! With a bit of help from people who know how to study meaning (hint, hint!), this type of research will be extremely valuable.

If anything, let this be an argument for computers & communication people working together, across disciplines.

In a future post, I will review conceptual and operational definitions of influence.

How to read a research article

Most research articles you find in academic journal follow a similar recipe. If you understand how the article is structured and what to look for in each section, you can read articles much faster. I can get what I want from a research article in 5 minutes or less. When I started grad. school it took me 45-60 minutes to get through a research article and I still didn’t get much out of it. I wish someone had taught me how to read them.

Here are my lessons, based on my experiences. They work for me. I hope they work for you, too. If they don’t, use this as a starting point to figure out your own reading process.

Understanding the anatomy of a research article will also help you write easier.

Title

Usually long and cryptic. Most titles are poorly written. I don’t pay much attention to the title.

Abstract

I read it carefully and look for:

  • purpose of study/research question
  • a hint as to research methods
  • key results

Introduction

I read the introduction looking for the following information:

  • explanation of the problem the study addresses
  • explanation of the larger context of the problem
  • argument about the importance/need/relevance of studying the problem
  • purpose of the study
  • an overview of how the article is structured, and how the next section is organized

Literature review

It may be called something else, or the article may not even have headings – but it should be there somewhere. The literature review should accomplish 2 purposes:

  1. make an argument for the need to conduct this specific study (identify a gap, or a need in previous literature)
  2. present the previous theories, concepts, etc. that this study uses and builds upon

Usually, each paragraph or small section of the literature review covers a body of literature (the best lit. reviews are organized thematically, IMO). When reading the literature review it is important to identify these major themes. They give you a lay of the land.

Imagine the body of literature is a garden. The article you’re reading attempts to plant a new seed in this garden. Before doing so, the authors explain the layout of the garden (vegetables here, flowers there, weeds over there) and they explain why their plant is needed and where it fits in.

When reading the lit. review, you get a feel for this garden. If you are:

  • very familiar with the literature, the lit. review confirms that the authors looked in all the right places and didn’t reinvent the wheel. OK to skim.
  • completely unfamiliar with the literature, this section will be terribly confusing. Don’t worry. All you have to get out of it are the major themes (sections of the garden). You can come back later and examine each individual plant. OK to skim.
  • are trying to learn the literature – read carefully, and mark on the list of references the sources you want to read.

The literature review ends with the research question(s). Find them and highlight them. They are promises that the article should deliver on.

Methods

This section explains the research methods and procedures used for the research study. Read them carefully, make sure they are valid. If the research methods are faulty, the data are not to be trusted. If the research methods are absurdly faulty, stop reading here. Go back to the literature review and the list of references and see if they can help you find better articles on the topic.

Results

In this section, the authors present their data, along with their (statistical or interpretive, etc.) analysis. This is as close as you can get to the raw data. This section, in a quantitative article, should be as free as possible of interpretation. Try your best to understand the results for yourself, so you can create your own interpretation of what they mean. But, if the statistics baffle you AND if you trust the authors, skim this section and move on to:

Discussion

This section explains what the results mean, in the context of the garden (literature review). You should see how the problem from the introduction is solved, how the research questions are answered, and whether the purpose of the study was accomplished. I usually read this section very carefully, because it tells me what the authors think they have accomplished.

Either here or at the end of the conclusion, you will find suggestions for future research. These can be very useful for your own literature review – you can cite the article, if it calls for exactly the research you’re doing. You can use this to support your own argument about the need for your research.

Conclusion

The first part of the conclusion should be a summary of the entire paper. I read it carefully, because the repetition helps me remember what I read. The last part of the conclusion is usually the most difficult part to write, very often fluff, and I don’t feel guilty about skimming or skipping it.

I used to teach this recipe to graduate students and they found it very helpful. I hope you do, too. Please share your own reading and writing tips, and ask me other questions you may have about graduate school.

There are several books that can help you, and the APA style manual has a chapter that explains the structure of APA research papers.

[update:] Barbara Nixon created a slide presentation for this content:

A College Student’s Recipe For Reading Research Articles

http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=acollegestudentsrecipeforreadingresearcharticles-090824105047-phpapp02&rel=0&stripped_title=a-college-students-recipe-for-reading-research-articles

Quantum Physics

Someday, I will understand quantum physics. But since in the past few weeks I’ve been unpacking, unpacking, unpacking, unpacking, unpacking… (you get it)… OK, never mind. Here’s a video about quantum physics. It should be the beginning of any research methods class.

Thanks to Twitter user @c4chaos for pointing to a link that lead me to this video.

“Utter bullocks”

I’m amused by this expression used in a comment on a RWW post titled: Study: 93% of Americans Want Companies to Have Presence on Social Media Sites.

I also believe it’s the perfect response to the report of this study, as presented in the RWW post. I don’t know, the study might be brilliant. But that’s the problem, they don’t provide enough information so I can decide if it’s brilliant or not.

Two issues here:

1) understand the data before you make decisions based on it

2) even if the data is good & valid, don’t jump in and make decisions based only on statistics & demographics

1) understand the data before you make decisions based on it

Some questions to ask about these particular results:

  • who are the 93% of Americans? There aren’t that many Americans online in the first place!!! (P.S.: cultural sensitivity issue: “America” includes Canda, and South American countries. Do you mean U.S. residents?)
  • they probably mean 93% of survey respondents, I guess (guessing = bad sign in research)
  • who are the survey respondents? Provide information about the sample:

These are just a few things I’d like to know before I’d spend a dime on a “social media presence”. And, as RWW writer Frederic L. points out, which social media sites? Twitter and Facebook are so different they might as well be two foreign countries!

2) even if the data is good & valid, don’t jump in and make decisions based only on statistics & demographics

My social media mantra is: It’s not about technology, it’s about culture.

Culture (social norms, etiquette, communication practices) emerges quickly around a social medium, and is specific not only to that medium, but also to sub-groups of users. So you can assume there are hundreds if not thousands sub-cultures on Facebook alone (about 100 million users worldwide).

An example: Befriending someone you haven’t met before is perfectly acceptable on Twitter, but creepy on Facebook.

So think about social media as a continent with many different countries and cultures. If you were to go to Romania (my native country), would you start doing PR & marketing armed with just some demographics produced by a poorly designed research study? I certainly hope not! I hope you’d take some time (a couple of years, say) to begin to get a grasp of Romanian culture before you dive in.

Same goes with social media. Start with your surveys, and make sure you understand what a good survey is. But do some ethnographic research, too (focus groups will do) before you spend that dime on your “social media presence.”

P.S.

Since I’m ranting, let me point out that the phrase “social media presence” is also … (see post’s title). It’s not about presence, it’s about engagement & conversation.