Institutional Repositories and measuring research impact


For better or worse universities and researchers are coming under increasing pressure to demonstrate the wider impact that their funded research has beyond the end of the research project.

But some are questioning whether traditional methods of measuring impact (peer-review, citation counts, Journal Impact Factor) are still fit for purpose in the modern research environment.

The open nature of the social web is continuing to have a disrupting effect on many aspects of scholarly communication. Today's research article is increasingly likely to be disseminated via social media/bookmarking websites making it instantly available to huge numbers of potential readers due to the innately viral nature of these services.

The exciting part is that modern web technologies make it possible to track each time an article is accessed from a tweet, blog or social bookmarking site. Now researchers can see in real-time how their research is being received around the globe. Transparent conversations evolve around papers as every (re)tweet, comment and annotation is available to be recorded and aggregated. This opens up the possibility to measure and define impact in new ways.

A number of emerging services are taking the first steps to build impact metrics based on these new usage data - collectively they are referred to as altmetrics. The people behind these services believe altmetrics may in future be used to supplement (or replace!) traditional methods of measuring impact - they even have a manifesto.

'View metrics'

These emerging altmetrics usually relate to the web-based use of the publisher's version of a research paper, so how can institutional repositories (IRs) capitalise on this broadening definition of research impact in order to benefit researchers?

The continuing failure to convince many researchers of the positive benefits of depositing to their IR is a common criticism of the repository community. The availability of this trove of rich usage data represents an opportunity for IRs to demonstrate the frequency with which the research they contain is used on the web.

Here at Manchester we've begun to make usage data captured using Google Analytics available through a new section of the service called 'View metrics'. Hopefully, by allowing researchers to see when, where and how often their eScholar records have been viewed and downloaded we can demonstrate how depositing research to eScholar can contribute to overall impact.

A screenshot of My eScholar - 'View metrics' 



In addition to usage data the 'View metrics' section also displays citation metrics under license from Thomson Reuters' InCites - Research Analytics as well as detailed metrics describing deposit activity. Researchers can view their own metrics as well as the metrics of their colleagues.

We're in the early stages of exposing impact metrics and it's difficult to predict how they will be used but we think there's value for the researcher and the institution in providing these metrics in a standardised way with a commitment to preserve the data indefinitely.    

For now the data displayed in 'View metrics' is available through an authenticated section of the service called My eScholar (requires University of Manchester log-on) but we'll soon start to expose usage and deposit data as appropriate on public-facing web pages.

If you're a member of staff or postgraduate research student at the University of Manchester then you can log-in to your My eScholar account now to check out 'View metrics'.

For further information visit our Frequently Asked Questions page or get in touch via email at escholar@manchester.ac.uk


Digital Curation

I recently gave a short presentation to colleagues from the John Rylands University Library. The theme was 'digital curation' and was structured around three questions. Here's the gist of it...

'What are the digital curation challenges facing Manchester eScholar?'

Manchester eScholar is populated through a self-archive deposit model whereby authors are encouraged to deposit metadata records (and where appropriate a copyright cleared full-text) with every new publication. 

This model has so far been successful. Over six thousand past and present members of staff have at least one record in eScholar and we estimate that the repository contains records for over 95% of the entire annual research output of the University over the past five years. 

To enable authors to deposit quickly there are very few barriers to submitting a record to the repository. This means no mandatory metadata fields and no checks by the repository support team. The model is successful in achieving high deposit rates but introduces data quality issues such as duplicated records, journal name variants, incorrect metadata...etc.

The strategic goal of the service is to sustain and enhance the research reputations of individuals and organisations affiliated with the University. Low quality metadata in the repository reduces the team's ability to achieve this goal; this is the major digital curation challenge facing Manchester eScholar.

'How are those digital curation challenges overcome?'

By establishing a programme of retrospective curation activities the team systematically improves the quality of the records stored in the repository. Examples of the types of activities include:

  • Correcting inaccurate metadata 
  • Deduplicating records
  • Adding additional metadata 
  • Adding Digital Object Identifiers (DOIs) 

We know, however, that when research is published it enters a dynamic ever-changing ecosystem. What we consider to be a high quality record today may not be considered so in five years time. Consequently, all records in the repository are placed in a 'perpetual curation cycle'. This means we will never stop curating a record!

'How can curating eScholar content benefit users of the service?'

A well thought through digital curation programme translates into the following benefits for users:

  • Increased 'findability' of the repository records on the web
  • Improved matching of external bibliometric data to repository records 
  • More accurate reports summarising total research output of organisations within the University
  • Better preservation of the records for future generations

We've also found that the more you can rely on the accuracy of the data in the repository, the more innovative ways you can think of to use the data. Mashing up repository data with data from external source allows for some really powerful insights into publishing activity.

And though it may sometimes feel like we're painting the Forth Bridge, the benefits make digital curation a vital  function of the service.

Here's the presentation...


SHERPA/RoMEO service gets upgrade

SHERPA Services launched a significant upgrade to its SHERPA/RoMEO service early last week.

Provided by the University of Nottingham, SHERPA/RoMEO lists the self-archiving policies of many of the world's academic publishers and has become a valuable and well-used resource in the move towards Open Access.

The layout and navigation of the site remains generally the same, albeit with a bit of a makeover. The real changes are in the journal data. Central to the upgrade is an new journal database containing data on 18,000 journals (taking new data from the Directory of Open Access Journals (DOAJ) and the Entrez journal list, in addition to the existing Zetoc journal data provided by the British Library).

The result is increased coverage and more accurate searching at the journal level. We've tried it out over the past week and can definitely see the improvements. It's now much clearer to see when a journal is published on behalf of a smaller academic society. And whereas before an ISSN search may have displayed policies for a number of publishers (i.e. Elsevier, Elsevier Cell Press) you now get a single publisher policy displayed per journal search.

We're impressed but see the Storify box below to see what others have been saying about the recent upgrade...

Manchester eScholar in Google Scholar

We were invited to talk at the University of Manchester's annual Research Support Conference yesterday about ways in which Manchester eScholar maximises access to the University's research.

As the title of the conference suggests, the majority of the delegates were either directly or indirectly involved in supporting academics here at the University in the creation, management and marketing of research. The key message of the talk was that once a record is created in eScholar and made open access we then do all we can to maximise the visibility of that record on the web.

During the presentation we demonstrated some of the core dissemination features of the service by showing how users can find eScholar content...

...directly via Manchester eScholar Search...
...or via the JRUL Vertical Search service...
...or via the main University of Manchester Search service...
...or via a link on the main University Researcher Staff Profile pages...
...or via a link on the School/Faculty Researcher Staff Profile pages...
...or (and perhaps most importantly) via Google.

However after the presentation a delegate asked us whether eScholar content was indexed by Google Scholar as this is clearly a very common way for researchers to find papers. Our answer was, 'No, but it should be.'

A quick search this morning revealed that many of the top UK Institutional Repositories do have their content indexed by Google Scholar and so clearly we'll need to act swiftly to ensure that our academics aren't disadvantaged for too long by having their eScholar records overlooked by the Google robots.

A look at Google's documentation suggests that the first two steps are to ensure that our web-pages are configured according to the Google Scholar inclusion guidelines and then to contact the Google Scholar team to be considered for inclusion.

We'll post again soon with further news.

Cloud Computing – Don’t Try This at Home!

I went to the Eduserv symposium last week – entitled ‘Virtualisation and the Cloud’. It was a well attended and well presented event which had many excellent speakers on a variety of topics concerning the state of cloud computing in the world at the moment. And while there were many interesting points made, two ideas struck me as very interesting.


The first idea was that building a genuine cloud computing solution should be left to the big guys – trying to create a cloud type solution on a small scale is fine as long as you don’t expect to reap the benefits that an industrial provider could give you. The principal of these is the ability to handle ‘burstiness’, i.e. short –term peaks in your system’s computing cycles or storage requirements. In order to handle this on a private cloud, you would have to invest large sums of money to handle a short-term problem – the very thing that the Cloud is meant to help with.


The second idea was that the amount of people using large-scale cloud solutions primarily for storage are being vastly overtaken by the number of people using the Cloud for computing cycles. The high overhead of data transport combined with fears of data security have seen Cloud users happy to have their computing and transaction processing being handled by the Cloud, but less happy to leave their data there.

So what does this mean to a data-centric sector like repository management? The answer might be a combination of using a private data centre whilst at the same time using a public Cloud provider to host new data as it expands past your storage limits. This then gives you time to assess the need for expansion of your own data centre – and proof to the person with the purse strings that you need it!

Spreading the word

Manchester eScholar went live in September 2009 delivering institutional repository (IR) services to The University of Manchester. Since then we've successfully embedded the IR into the publication workflow for the majority of our researchers, in the process achieving virtually 100% deposit rates for all research outputs created since 2005.

Of course we can't be complacent; the web doesn't stand still and nor does the scholarly communication landscape. To sustain this high level of engagement we know we must deliver new and useful functionality whilst continuing to improve existing services.

Whilst we're confident that we're on the right road to delivering high quality IR services we recognise that it's all for nothing if we don't let people know about what we're doing. So we sat down and wrote a Manchester eScholar Communication Strategy. Our aims for the strategy are to:

  • sustain and enhance awareness of Manchester eScholar Services within our institution's research community
  • promote support for scholarly communication across our institution
  • increase the exposure of Manchester eScholar to external audiences including the global repository and open access (OA) communities

Institutional repository's have different categories of users so within the strategy we've identified a number of different communication channels. Establishing communication channels alone won't guarantee that the right message reaches the right audience; we have to commit to regular and frequent communications and to get into the habit of thinking, 'Would that make a good blog?', or, 'I must tweet that!'.

In line with our communication strategy here are some of the key communication activities that we'll be undertaking over the coming months:

Establish this blog

We'll use this blog to announce new releases and bug fixes but we'll also use it to write about the hot topics in open access and digital repositories. And we won't be too hurt if our readership doesn't extend into the thousands, the process of writing down our thoughts and opinions should be a useful exercise in itself.

We all know the blogosphere is littered with abandoned blogs that haven't been updated or visited for years. To prevent the same fate befalling this blog we aim to post at least once a month and we are encouraging all eScholar team members to contribute.

Establish a Twitter account

There are plenty of active Twitterers in the global repository community so hopefully by starting an account we'll be able to join in with some of the really interesting conversations going on out there. It's also a good way to keep up to date with the latest news and events. And anyway no communication strategy would be complete these days without a Twitter account. Our Twitter name is @_escholar so please follow us.

Establish a Manchester eScholar email mailing list

This mailing list will just be for our colleagues from The University of Manchester. Hopefully the posts will be useful and interesting for staff who provide scholarly communication support to researchers or those who want to keep up to date with the latest developments in OA and digital repositories. By establishing this channel of communication we hope to more effectively cascade key information out across our institution.

'Showcase' research publications

We've already begun to liaise with our colleagues in the University's Communications, Media and Public Relations division to showcase the noteworthy publications stored in Manchester eScholar on University websites. Whenever a research output from the University gets 'media buzz' we want to ensure that links are in place to point people towards a high quality Manchester eScholar record containing full metadata and a copyright-cleared version of the full-text.

As well as the above activities we'll also be running Manchester eScholar workshops throughout the year, submitting an article to UniLife, the University of Manchester's monthly glossy magazine, and hopefully publishing our work at national and international events and in peer-reviewed journals. So plenty to be done!!

We'll be regularly reviewing our progress as we move through the year and asking ourselves how effective our activities are being and what we might be able to do to be more effective. Only time will tell. So stay posted.

Welcome

Welcome to the Manchester eScholar blog. Manchester eScholar is The University of Manchester's institutional repository and is made up of a suite of services.
The Manchester eScholar Support Team - from left to right, Phil Butler, Nilani Ganeshwaran, Scott Taylor, Thom Grahame and Jon Besson
Manchester eScholar Services is supported by a small team of individuals hosted by John Rylands University Library. Our mission is to
  • sustain and enhance the research reputations of individuals and organisations affiliated with The University of Manchester
  • enhance the global research community's ability to access The University of Manchester's research outputs
This blog is to share what's new with Manchester eScholar, digital repositories and scholarly communication and our experiences of supporting a repository service for a large university. Example future articles will cover, our communication strategy, aspects of our digital curation activities, technical stuff, briefings on workshops and conferences attended, and our approaches towards research data management.
So watch this space.