How do we measure the impact of research? And how can we learn to do it better?

From September 12 to 14, the ScholCommLab’s Asura Enkhbayar will attend STI 2018, an international bibliometrics and scientiometrics conference dedicated to answering these questions. Hosted by the Centre for Science and Technology Studies (CWTS) at Leiden University in collaboration with the European Network of Indicator Developers (ENID), the conference will cover a diverse array of topics from reproducibility in scientometrics to evaluation of open scholarship and everything in between.

Here, Enkhbayar tells us more about the conference, the research he’ll be presenting, and what he’s most looking forward to about visiting the beautiful city of Leiden.  

Tell me about STI. What’s the focus of the conference?

The STI is the biggest conference for scientiometricians and bibliometricians. We are attending especially because of the track dedicated to challenges of social media data for bibliometrics.

The theme of this year’s conference is “indicators in transition” and their role in driving more comprehensive, socially oriented forms of Science, Technology and Innovation evaluation. What, in your opinion, are some of the most important “indicators in transition” we’re seeing today?

That’s a big question! Well, I guess it’s the big question that everyone’s trying to answer at the moment. There’s a large demand for—I’ll just put it this way—better indicators, because people are spotting problems with the current status quo.

Some years ago, the big focus in this field was altmetrics, which is supposedly a better indicator of social impact. But, now, with all the big developments in social media and digital communication, people are critically investigating the nature of many new indicators. It’s an ongoing development, and I feel like there’s a new critical perspective coming into play, which is being emphasized more and more in research. I think that’s a great thing.

So I have no concrete answer for you, just support for the questions around new indicators that researchers are starting to ask.

You’ll be presenting some work you did with the ScholCommLab about Facebook metrics. Can you tell me, briefly, what it was about and how it fits with STI?

It was an obvious choice to submit our paper to the STI this year. The conference, and its focus on technical challenges, is a perfect fit for the research I’ve been doing with Juan Pablo Alperin at the lab. In our study, we looked at Facebook as a data source for scientometrics. We wanted to figure out, in the beginning, how different approaches for collecting engagement data would influence the results of various the collected metrics. But in the process of doing this research, we uncovered a number of technical challenges within altmetrics that make this kind of work difficult. We realized that a very big chunk of the work we had to do was about fundamental technicalities that needed to be figured out in order to get to reliable data. We ended up investigating some of those challenges instead.

So what was supposed to be a quick project turned out to be months of work on technical details related to the general structure of the web and scientific publishing, but also to Facebook’s internal algorithms, which are still a black box for us.

So Facebook data could have the potential to be a really good indicator, but all of these technical issues are getting in the way?

In this particular approach, yes. As always, it depends on many things, but it’s important to figure out these details before using any new indicator. This is still a work in progress, but we are trying to answer the question: How much do these technicalities affect the outcomes? I think it’s a very essential question to address, if one uses Facebook as a data source.

So the next step in your research will be to investigate and understand whether, or to what degree, these technical difficulties affect the outcomes of research that relies on Facebook data?

Exactly. We have now finished the first part of the project, and we’re finally starting to look into the questions that we were initially hoping to answer. We’ve been sidetracked for a long time—well, more than sidetracked, because the challenges that came up were really important—but I’m looking forward to taking this research further.

What are you most excited about seeing or doing at the conference?

I’m really looking forward to some of the presentations about the exact same topic we’ve been investigating, to figure out what other people are doing to overcome these challenges. Because I think these are questions that have been surfacing for many researchers recently.

But obviously, the conference is bigger and broader than that. I recently attended the CWTS summer school on scientometrics, so I’ve had my first overview and introduction to the field. Now that I have this formal education to build off of, I’m really looking forward to learning more about some of the research that’s being done in this area: seeing the projects and the people doing the work, and really getting a sense of what’s going on in the field.  

Let’s end with one fun question: is there anything you’re really looking forward to doing in the lovely city of Leiden?

I guess what I want to do is go on a proper pub crawl: finding a few bars, trying a few beers. The first time I was there, I didn’t know the city, I didn’t know the people. But now that I’ve already been there once, I feel like I’m familiar with the basics of the city. This time, it will be different. This time I’m ready.

For more information about the conference, visit sti2018.cwts.nl. Read more about Enkhbayar’s research at arXiv.org.