Category Archives: Evaluation Reports

In June 2011 we came to the end of a grant from the Wellcome Trust to run I’m a Scientist in 2010 and 2011. We’ve evaluated throughout the project and the final evaluation report is available to download below, as well as a shorter executive summary and summary sheet.

2010-2011 Evaluation Report – at 75 pages long it contains a lot information including the methodology, aims, key figures and data, findings and recommendations.

Executive Summary – this outlines the main findings from the 2010-2011 Evaluation Report and looks at whether the objectives set at the start of the project have been met.

Summary sheet of the event in 2010-2011 – this highlights some of the key numbers and quotes from evaluations, and reasons why the event works so well.

How do we evaluate?

Evaluation is absolutely key to understanding what students and scientists have got out of the event, and it helps us continuously improve the event. There are two types of evaluation that we’ve done on I’m a Scientist, right from the first pilot event back in 2008.

Formative: We have done formative evaluation throughout the project – asking students, scientists, teachers and stakeholders for their views and acting upon them.

Summative: This is the more formal type of evaluation where we ask participants for their views before the events starts, observe the event in action in the classroom and survey views after the event has finished.

One of the benefits of running an online event is that we collect an enormous amount of data about how students and scientists engage with the event. We know how many students asked questions, how many questions they asked, how much they participated in live chats. We can tell the scientists how many students they’ve reached, how many questions they’ve answered and whether they’ve changed the attitudes of those students.

We’re always looking for ways to improve our evaluation, but what’s the point of evaluation is no-one reads and learns from what you’ve done. We’re genuinely interested in hearing what measures you’d like to see. What is it you want to see in an evaluation? Send us an email with your suggestions.

Previous evaluation reports

You can find the evaluation report summary event here

You can find the pilot evaluation report here

You can find the complete evaluation report on the 2008 pilot event here (this is the full, 127 page long baby)

You can find a summary of the 2008 pilot event evaluation report here

In our pilot event, we did nearly all of the evaluation in-house, to save money. We tried to be as objective as possible, but of course it’s much better practice to have an independent evaluator. This year, we’ve been able to do that, and we appointed the lovely Kate Pontin to evaluate the project for us.

We are publishing the evaluation report on 2010 today (and also presenting it at the Science Council). You can also read the team’s early thoughts on this year in our write up of our Science Online London session, or see our evaluation report on the pilot.

Download full I’m a Scientist Evaluation Report 2010 (pdf)

Summary of I’m a Scientist 2010 evaluation

This report summarises the findings of the first year of formal evaluation of I’m a Scientist, Get me out of Here! and thus acts as an interim report.

I’m a Scientist creates opportunities for scientists to answer questions from students via the internet, opening dialogue, while developing skills and understanding of the science process. It also gives students an idea of what it is like to be a scientist and the types of careers available.

Evaluation data was collected from scientists, teachers and pupils via surveys, and interviews. Observations of a number of sessions in school were also undertaken.

Data suggested that the IAS programme is very successful, providing a unique approach to the communication of science to students. It shows that:

• Students gained a lot from the experience. Their feedback shows they thoroughly enjoyed the event, especially the live chat sessions. They developed key skills including focused questioning and gained confidence in scientific discussion. They found it interesting to discover more about what it is like to be a scientist and were surprised that scientists were not as stereotypical as they originally thought.
• Teachers found that IAS supported their need to develop different approaches in the classroom and also to encourage pupils to think about science and who it is actually undertaken by. They found the resources very useful, in particular the debate kit and were also pleased with the support they received from the team during the running of the event.
• Scientists also gained or developed skills in communicating with the public, explaining their work (often complex science) clearly to pupils aged 13 onwards. This in some also helped inspire and revive their enthusiasm for their own studies.

Minor improvements suggested are:

• Helping teachers to prepare pupils especially in their development of clearly focused questions
• Promote and signpost other debate lesson packs like the IVF debate, and to help conclude the event, with perhaps an assessment element.
• Further initial guidance for scientists giving guidance on the time it might take, but also for those less familiar with students of this age some general information on KS3/4 and the diverse knowledge base.

Evaluation will in 2011 focus on finding out more about how to attract scientists and about the impact their involvement has on their organisations. Longitudinal studies will look at the impact of students in the longer term (for example in take up of science or exam results). It is also hoped to develop baseline and plenary activities to find out more about change in knowledge and attitude to science.

I’m a Scientist enabled students using an innovative approach to inspire and enthuse. It worked with pupils from high achieving classes but also those across the ability range and from a wide range of backgrounds including groups from BME families.

Key figures for 2010

• Two events – March and June
• 7,459 questions asked
• 125 scientists were involved, in 25 zones
• 94,909 visits to the site
• 6397 students took part
• 648,563 page views

About the evaluator

Kate Pontin is a former geologist who has been working in museums and evaluation for over 20 years. She specialises in the experiences of young people in informal learning environments.