ImpactStory: Telling Data-Driven Stories About Research Impact

ImpactStory is the relaunched version of total-impact. Don’t miss the post by @jasonpriem and Heather Piwowar, published on the Impact of Social Science blog. They describe some of the highlights:

To use ImpactStory, start by pointing it to the scholarly products you’ve made: articles from Google Scholar Profiles, software on GitHub, presentations on SlideShare, and datasets on Dryad (and we’ve got more importers on the way).

Then we search over a dozen Web APIs to learn where your stuff is making an impact. Instead of the Wall Of Numbers, we categorize your impacts along two dimensions: audience (scholars or the public) and type of engagement with research (view, discuss, save, cite, and recommend).

In each dimension, we figure your percentile score compared to a baseline; in the case of articles, the baseline is “articles indexed in Web of Science that year.” If your 2009 paper has 17 Mendeley readers, for example, that puts you in the 87th-98th percentile of all WoS-indexed articles published in 2009 (we report percentiles as a range expressing the 95% confidence interval). Since it’s above the 75th percentile, we also give it a “highly saved by scholars” badge. Scanning the badges helps you get a sense of your collection’s overall strengths, while also letting  you easily spot success stories.

 

How Social Media Is Changing the Way We Talk About Science

Five Questions With UCSF Neuroscientist Bradley Voytek

Brad Voytek, PhD, a post-doctoral fellow at the University of California, San Francisco, makes use of big data, mapping, and mathematics to discover how brain regions work together and give rise to cognition. In his work as a researcher, science teacher, and outreach advocate, he regularly uses social media such as his blog Oscillatory ThoughtsTwitter, and Quora. In 2006, he split the Time magazine Person of the Year award.

Bradley Voytek

Brad Voytek, PhD

Q: You’re interested in leveraging data to modernize science. Do you see a role for social media in changing research? 

Social media already is changing research. In many ways! First, there’s a direct effect wherein scientists are beginning to use data collected by social media organizations to analyze behavioral patterns, such as a study from 2011 that looked at millions of tweets to analyze fluctuations in mood.

The way we conduct and communicate research is also changing. Since the 1600s, scientists have communicated findings through peer review. But these results are static, and conversations regarding specific study details, methods, etc. were private. Now, scientific publishing organizations like Nature Network or SciVerse and professional sites such as Mendeley and ResearchGate are providing platforms for open communication and ongoing conversations about research projects.

Q: You’re using social media to promote your work. What motivates you to do that, and what do you see as the benefit?

While in a sense the first statement is correct, “promotion” is a loaded term. Science is an opaque process, and scientific publications are jargon-laden, dense documents that are inaccessible to all but the field-specific experts. These publications give an idealized view of the scientific process–from clear hypothesis to statistically significant result. The reality is a world messier and less certain than that, and I use my blog to communicate that.

Having students just jump in and read these artificially-refined and specialized manuscripts and Social media quote_Bradley Voytekexpecting them to learn from it is like trying to teach English by having someone read Shakespeare: it’s technically correct but the end result will be a mess.

I get a sense that many of my blog’s readers are undergraduate and graduate students, and I aim to communicate the real difficulties and uncertainties of science with them. I remember being there feeling confused, and feeling really dumb because I didn’t “get” scientific papers and could never imagine myself coming up with a novel idea, running an experiment to test it, and writing a paper. I remember looking at the CVs of really smart post-docs and professors and seeing page after page of amazing compliments and thinking I was inadequate.

My goal is to demystify the scientific process, to make it more real, to show how hard everything is, but also that it’s doable. I’ve got a whole section of my CV titled “Rejections & Failures” outlining every grant or award I was not given, every paper not published. I believe that listing those failures shows fledgling scientists that the process is hard, but not because it requires super-intelligence, but rather super-diligence.

Q: You recently made an offer on Twitter inviting people to ask you questions about neuroscience. Can you tell us about that?

This exemplifies another reason I blog, use Twitter, etc. When teaching, I took to heart the idea that if you can’t explain something clearly, then you truly have not internalized it and don’t really understand it.

Social media is a way for me to continue sharpening my understanding of difficult concepts. The time investment isn’t important to me–my job is to learn and discover, and this is another aspect of that. And if in the process I make something more clear and accessible to a possible future scientist, all the better. No scientist achieved their breakthroughs because they communicated less.

As for the offer on Twitter, I got quite a number of excellent questions, but a few stood out that really made me think. Specifically, there were three that are directly relevant to my research and that got me digging around the literature some more to figure out the answer. The questions essentially boiled down to two ideas: First, how plastic is a mature brain? And second, how many neurons can you lose before you (or someone else) notices?

Ultimately I wrote a blog post on what I’d found and rolled some of that writing and those ideas into peer-reviewed papers I’m still working on. This kind of challenge, discussion, and ideation exchange is extremely valuable for me, and it’s part of the reason that I make offers such as the one on Twitter or use Q&A sites such as Quora.

Quora is a particularly interesting example. It’s a site populated by very intelligent people, but given the kinds of neuroscience-related questions that appear there, it’s clear that there are still some pervasive misconceptions about how the brain works. On a site such as that, the feedback and discussions seem to flow a bit more easily than they do on my own personal blog, but they’re not limited in scope as on Twitter. It’s a nicer platform for the level of discussion I’m seeking.

It also doesn’t hurt my motivation when I get comments from people such as, “I’m a grown man with a family and a career and [Brad] made me want to become a neuroscientist!” or “I accidentally started liking science stuff thanks to you!”

Q: Lots of scientists are not using social media. When asked why, many say they don’t think people will care about their scientific work. What do you think about that perspective?

People who say such things underestimate the interest level and intelligence of the non-scientist public. When I hear this, in my head it translates to either, “I don’t care about what I’m doing,” or, “I’m not confident enough in what I’m doing to explain it to anyone who may ask really simple questions that undermine what I do.” The former is fine; not everyone needs to “love” their job or work to be excellent at it. The latter is emblematic of unclear thinking.

Q: What tips can you give researchers who are thinking about using social media but don’t know where to start?

Generally the tips I’ve seen from a lot of bloggers are “write consistently” and “be engaging”, but that’s like saying to be a good scientist you need to “work harder and be smarter”: technically true but not very useful. I wish I had some magic formula for how to be successful at using social media for science, but I don’t have such a thing.

Broadly speaking, knowing how to communicate complex ideas effectively is critical, but just as important is knowing how to network, how to spread your ideas, and how to write something other people want to read. You’ve got maybe a few seconds to capture peoples’ attention online, and getting them to read a 1000-2000 word article is hard. Time and attention are premium commodities in people’s lives, and what you’re asking them to do is sacrifice that commodity to you. You have to keep that in mind. When you write, don’t think “this will only be read by a half dozen of my friends who read my blog.” Instead, think, “this might get picked up and read by tens of thousands of people. Is this worth the time of thousands of people?”

I find social media helpful to clarify my thinking, but other people may have other methods of accomplishing the same result. The only remaining advice I have is to seriously consider the reasons for not using social media: are you not blogging/tweeting/whatever because you honestly think it’s a waste of time and can see no return-on-investment for you? Or, are you not doing it because simplifying your ideas is too challenging?

This Q&A is part of “Digital Media & Science: A Perspectives Series from CTSI at UCSF” and was originally published on the UCSF CTSI website. This series explores how digital media and communications can be used to advance science and support academia.

Related posts by Bradley Voytek

Brad is also interested in leveraging data to modernize research. He’s one of the creators of brainSCANr, an online resource that uses existing publication data to show the probability of relationships between neuroscience topics and ultimately support the discovery of novel research ideas. He is also a fan of zombies, and has devoted some of his time to mapping brain damage that would be caused by zombification.


Describing the Difference Research Has Made to the World

Here is an interesting new blog post by Heather Piwowar about the different ways research can impact the world and the importance of telling them apart. Good food for thought as we think about ways to help researchers analyze how people are reading, bookmarking, sharing, discussing, and citing research online.

I think Anirvan made a great point to think about ways how we can integrate “altmetrics” data with UCSF Profiles. Some of the metrics mentioned below may be a great starting point.

Here’s what Heather writes:

Figuring out the flavors of science impact. CC-BY-NC by maniacyak on flickr

We have clustered all PLoS ONE papers published before 2010 using five metrics that are fairly distinct from one another: HTML article page views, number of Mendeley reader bookmarks, Faculty of 1000 score, Web of Science citation counts as of 2011, and a combo count of twitter, Facebook, delicious, and blog discussion.

We normalized the metrics to account for differences due to publication date and service popularity, transformed them, and standardized to a common scale. We tried lots of cluster possibilities; it seems that five clusters fit this particular sample the best.

Here is a taste of the clusters we found.  Bright blue in the figure below means that the metric has high values in that cluster, dark grey means the metric doesn’t have much activity.  For example, papers in “flavour E” in the first column have fairly low scores on all five metrics, whereas papers in “flavour C” on the far right have a lot of HTML page views and Sharing (blog posts, tweeting, facebook clicking, etc) activity. View image and read on

Further reading:

Crowdsourcing the Analysis and Impact of Scholarly Tweets

“Twitter is one of the fastest tools to discover newly published scholarly papers”, Martin Fenner wrote in one of his earlier posts. Now Fenner and Euan Adie finished the first phase of an interesting new experiment, the CrowdoMeter project.

In the last 2 months, they used crowdsourcing to analyze the semantic content of almost 500 tweets linking to scholarly papers (953 classifications by 105 users for 467 tweets). Their preliminary results show: 

  • 3 predominant subject areas: Medicine and Health, Life Sciences, and Social Sciences and Economics.
  • Most tweets (88%) discussed the papers, 10% were in agreement, 3% disagreed.
  • Most papers are not tweeted by their authors or publishers

In his recent guest post on Impact of Social Sciences, Fenner makes the argument that “social media, and Twitter in particular, provide almost instant, relevant recommendations as opposed to traditional citations.

A few years from now the ‘personalized journal’ will have replaced the traditional journal as the primary means to discover new scholarly papers with impact to our work.

What is still missing are better tools that integrate social media with scholarly content, in particular personalized recommendations based on the content you are interested in (your Mendeley or CiteULike library are a good approximation) and the people you follow on Twitter and other social media.

Fenner’s view is also based on Gunther Eysenbach’s study from 2011 that showed “highly tweeted papers were more likely to become highly cited (but the numbers were to small for any firm conclusions; 12 out of 286 papers were highly tweeted)”.

Fenner and Adie are using altmetric.com to track the scholarly impact of your research. Other tools – some of which we wrote about – include  ReaderMeterTotal ImpactPLoS Article-Level Metrics, and ScienceCard.