The challenges of online research

It’s exciting to see how much information is readily available on the Internet, especially on a topic like social media.  I am delighted to have the opportunity to analyze and synthesize data from so many sources. However, as often happens when data are simplified for presentation to the public, some of the numbers are presented without enough context, which makes it difficult to interpret them. As I compared across web sites, I started to notice places where the numbers didn’t line up. This reflects the fact that usage and trends in the digital world change constantly, while research reports stay online, trapped in the moment when they were most recently uploaded. Social media usage world-wide is growing rapidly. So numbers from March of this year are significantly lower than numbers from September of this year. I have encountered sites reporting that MySpace has more members than any other social networking site, and sites reporting that Facebook has this honor. This is because Facebook’s growth skyrocketed, and they passed MySpace globally in April of this year (MySpace still has more members in the US).

Another issue is that differences in methodology can lead to different results. In academia, research reports generally include detailed accounts of their methodology, the sample they use, the wording of the questions asked. This is great for hard core researchers who want to know every detail of how the data were gathered (and use those details to gain a better understanding of what the data really mean), but is unlikely to hold the interest of the average blog-surfer. So write-ups for the digital audience tend to be more concise, and generally skim over much, if not all, of the specific information about methodology (sample, data collection, etc).

Yesterday I read Universal McCann’s Wave.3 report on social media, which asserts that 58% of Internet users globally have created a profile on a social network. So I decided I should check some of the Social Media posts on our own Molecular Voices. I quickly found Yuval Zukerman’s recent post on lack of awareness of social networks. He cites a well-designed study by Molecular’s sister company Synovate, which reports that 58% of global users don’t know what a social network is.

What’s going on here? These are both large-scale studies (Universal McCann surveyed 17,000 users world-wide; Synovate surveyed 13,000), from well-respected companies with a history of doing quality research. So how do we get to a situation in which 58% of users have created profiles on social networks, while only 42% even know what social networks are?

I can think of two reasons off the top of my head why this might be so.

1) They may be defining “user” differently: Wave.3 explicitly states that they surveyed only people who use the Internet every day or every other day. Neither Yuval’s report, nor the summary posted by Synovate articulate who their “users” are. If Synovate is using less stringent criteria (e.g., they count as users anyone who’s logged on in the past month, or anyone who has an email address), they’re looking at a larger population than Universal McCann.  Synovate’s entire population could be so much larger than Universal McCann’s that 42% of the Synovate population includes more people than 58% of Universal McCann’s population.

2) Language about online phenomena is ephemeral, and not universally adopted. Many people who use networking sites may not know they are using “social media,” not unlike M. Jourdain in Moliere’s The Bourgeoise Gentilhomme, who is surprised to learn at age 70 that he has been speaking prose all his life. Neither report gives full wording of the question, but it’s possible that the wave.3 report’s question was along the lines of “Have you ever created a profile on a social networking site, such as Facebook or MySpace?” while Synovate’s respondents were merely asked “Are you familiar with social networking sites?”

This problem becomes more intense as you begin to look at third party reporting of research (such as blog posts or e-news articles describing someone else’s work). Last week a large number of news and blogging sites discussed Bill Tancer’s new book, Click, largely focusing on one finding: social networking sites have surpassed porn sites in popularity. This is an intriguing finding, but I can’t find in any of the articles the specifics about how popularity was measured. Did Tancer compare the number of unique visitors to each site? The amount of time each visitor spends on the site? The number of links to each site? Without this information, it is difficult to tell what the statistics mean. Many of the articles talk in detail about search, suggesting that more people are searching for social media sites than for pornography. None of the articles indicate whether the data is from click streams or self-report. If it’s self-reported, I have difficulty accepting it: people are more apt to report engaging socially acceptable behaviors. I could probably find the answers to these questions in Tancer’s book. However, I suspect that far more people will encounter this information on line than will read Click.

Thanks to the Internet, we have access to more numbers, graphs, and trends than ever before. The proliferation of information available to all of us now requires that we be thoughtful consumers of such information. Numbers take on different meanings in different contexts. In situations where we can’t learn the context, we should take care in interpreting the data. And in situations where we can take the time to provide context, we should do so. This enables us to develop a richer, more complex understanding of the phenomena the data represent.