ACU Measuring Success? Blog on longitudinal scholarship research

Re-post of my recent blog in the ACU’s Measuring Success? series. Original post here: ACU Measuring Success?


Tracking international scholarship outcomes: the CSC Longitudinal Research Framework

Matt Mawer, Rachel Day, and Shireen Quraishi (Commonwealth Scholarship Commission in the UK)

In stark contrast to only five years ago, much of the prominent new research on scholarship programme outcomes adopts a longitudinal model. Some examples have been posted within this blog series: see, for instance, essays by Anna Seifried and Cate Lawrence or Mirka Martel. A recent request for proposals from the MasterCard Foundation – architects of the Scholars Program – reinforces the point:

“The Scholars Program Longitudinal Cohort Study will be a central component of the Program’s learning partnership. It will seek to track post-Program Scholar outcomes and to understand the impact of Scholars on their families, communities, organizations, and societies.” (RFP 3, 4th March 2017)

The increased profile of longitudinal research within scholarship evaluation is a testament both to the difficulties presented by previous approaches and to the will of policymakers and administering agencies to grapple with those complexities and strengthen the evidence-base for grant making. Whether conducted within administering agencies or outsourced to expert consultants, contemporaneous research tracking scholarship alumni during and post-scholarship is fast becoming the ‘gold standard’ for assessing individual-level impacts.

Like others in this field, the Commonwealth Scholarship Commission in the UK (CSC) has now invested considerable time and energy into building an effective longitudinal study of mid-range scholarship outcomes (professional development, organisational change, etc.). Unlike most others in the field, however, the CSC Longitudinal Research Framework is designed to continue perpetually as a component of basic scholarship administration. We introduce the logic and methodology of the framework in our article. Continue reading

Article published: Approaches to analysing international scholarship programmes

To start the new year, the Journal of Studies in International Education has published my paper: “Approaches to Analyzing the Outcomes of International Scholarship Programs for Higher Education

The paper is based on research I conducted at the Commonwealth Scholarship Commission examining published literature on the outcomes of scholarship programmes and focuses on three issues: 1) the relationship between aims and outcomes, 2) difficulties with “attribution” and “contribution,” and 3) scholarship programs in comparison with their alternatives. You can also read the original report at the CSC website.

Continue reading

‘Measuring Success?’ blog – Editorial

The end of October 2016 marks the one-year anniversary of ‘Measuring Success?’: The Association of Commonwealth Universities’ blog series about research on scholarship outcomes. To celebrate a year of fascinating content from authors around the world, I have written a reflection with my co-editor Sian Julian:

‘Measuring Success?’ year one – The scene, the motifs, the actors

The blog series has been developed – or perhaps curated – without a strong editorial line. We have helped develop some themes, but have not guided the topics much beyond defining the basic parameter that posts had to be relevant to understanding the outcomes of scholarship programmes. Reflecting on the contributions across 2016, it is evident that the series has grown to be much more than simply a technical blog about methodology (if it even started as such!)

Measuring Success will continue into 2017: we already have a few pieces lined up and plan to start the new year with a mini-series of linked posts. As always, we are happy to receive ideas for content and volunteers to write for the blog: send me an email directly or use the email link on the blog landing page to get in touch.

In review: Measuring Success? blog series, 2015

At the end of 2015 the Measuring Success? blog series has begun to pick up pace. Four contributions have been posted so far, with several more agreed with authors for early next year. So far we have examined:

Geographical mobility and the Rhodes Scholarships across the 20th century
Authored by Dr Tamson Pietsch, published on 16/10/2015
The Rhodes Scholarships scheme is one of the longest running programmes of scholarly exchange, and as such it offers a century’s worth of invaluable data for examining patterns in the geographical mobility of awardees post-scholarship. The author’s analysis has revealed some striking patterns. Read more

International scholarships and the SDGs
Authored by Dr John Kirkland, published on 06/11/2015
Half a cheer for the new Sustainable Development Goals (SDGs). On the one hand it’s good to see increased prominence given to tertiary education, including a welcome reference to scholarships. But problems will emerge from the targets in themselves. Read more

Methodologies for evaluating short fellowship programmes
Authored by Dr Caroline Cage and Shireen Quraishi, published on 30/11/2015
Short fellowship programmes offer academics a tailored opportunity to develop professional skills. Given the individual nature of the fellowships, evaluation presents some challenges. Here, the authors outline the methodologies in place to assess impact of two specific schemes administered by the ACU. Read more

Defining success in light of student experiences
Authored by Aryn Baxter, published on 18/12/2015
A reflection on how the experiences of international scholarship recipients from low-income contexts might inform the ways we choose to define and measure success. Read more

If one can judge by the positive feedback received on the initiative, it seems that a space for short, critical pieces on topics highly salient to scholarship programmes is a welcome addition to the research, programme administration, and policymaking landscape.

‘Measuring Success?’ blog launches

Last week we launched a new blog series called ‘Measuring Success?’, co-curated by The Association of Commonwealth Universities (ACU) and the Commonwealth Scholarship Commission (CSC).

Measuring success banner v1 plain blue text updated cropped

The first contribution to the series is now live on the ACU Voice blog:

Geographical mobility and the Rhodes Scholarships across the 20th centuryDr Tamson Pietsch, University of Sydney

You can also follow the series using its tag on the ACU website, just click on the banner above or in the sidebar. Continue reading

Tracing scholarship outcomes at EAIE Glasgow 2015

The week of 15-18 September 2015 saw the 27th annual conference of the European Association for International Education (EAIE) held in Glasgow, UK. I joined a panel of colleagues from EP-Nuffic and DAAD to discuss ‘Tracing the outcomes of study abroad scholarships’ and the potential for deeper collaboration between agencies in future.

Evaluation timeline

A potted version of the paper is available here on the CSCUK website.

Debating the future of international scholarships at NAFSA 2015

How can we understand the outcomes of international scholarship programmes? And what shapes the long-term impact of programmes on their alumni and society? These were the central questions posed to a panel of experts at the NAFSA Association of International Educators annual conference in Boston, last month.

After the conference I published a few thoughts on the ACU’s website.

Article published on observation in virtual worlds

After some time in the publishing mill my paper on observational research in virtual worlds (such as Second Life) is now available in the International Journal of Social Research Methodology.

The paper tackles some technical issues relating to conducting field observations in virtual worlds and might be of interest to those who (like me) have used, or are considering using, the approach as part of their research.

Continue reading

Evaluation at the Donor Harmonisation Group

Each year a network of organisations administrating higher education, research, and capacity building development cooperation programmes meets in Europe to share insight, collaborate, and, where possible, align objectives. The ‘Donor Harmonisation Group’ this year met on June 11-13 in Helsinki for their annual forum, hosted by the Finnish Centre for International Mobility (CIMO).

Donor Harmonisation Group, Helsinki, June 2014

Donor Harmonisation Group, Helsinki, June 2014

Continue reading

Evaluation methodology study published

‘A study of research methodology used in evaluations of international scholarship schemes for higher education’

Over the last six months we have been working on a scoping study examining the state of research practice in the sector: trends, ambiguities, and omissions. The aim has been both to inform our own evaluation practice and to update our colleagues around the world on how evaluation is being conducted.

The final report is now available on the CSCUK website.

Continue reading