Category Archives: Evaluation Reports

GM Food Zone Evaluation

GM Food Zone

For some time we’ve been wanting to expand the I’m a Scientist format and allow the general public to participate. The hysteria generated over the Rothamsted GM Wheat trials in May 2012 presented us with the opportunity to run a GM Food Zone.

We’ve written up our evaluation of the project. There are 4 sections in addition to this page:

You might find this incomplete. Please ask more questions in the comments.

For those who are too impatient to read the entire evaluation here is

The short version…

Driven by some hysterical writing in the national press we decided to run a special I’m a Scientist GM Food Zone and open it to the general public. We wanted to create a neutral space where a constructive and civil discussion could take place.

We knew that we were placing ourselves in the line of fire. We knew we had to be neutral and seen to be neutral and balanced. We recruited an advisory panel and asked for nominations for the experts who would answer the questions. We had 70 nominations and despite an enormous amount of work to ensure we had the right range and balance of experts we were still accused of being Pro-GM. It’s possible that assumptions were made because of the I’m a Scientist name or the 3:2 ratio of experts in favour of GM.

As the questions came in they were fairly balanced. Slightly more were anti-GM than Pro-GM. Almost half the questions were from teachers and school students. A different set of visitors then started leaving comments. Three determined anti-GM visitors left just over half the comments we received. There was very little cross-over between the group of people asking questions and those leaving comments.

The site has so far been visited 3,977 times by 2,370 unique visitors. They averaged 3.4 pages per visit, but most pleasing is the stat that on average visitors spent 3:19 min on each of the 5 most popular questions. Visitors were spending a reasonable amount of time reading through the answers and comments.

UPDATE 09/01/15

7,900 visits from 5,900 users.

So was the GM Food Zone successful?

Yes. We did create a space where discussion could take place. The discussion was polite and mostly respectful. Many people read the discussions and a small group contributed towards it. Whilst it would seem that comments tended to be left by people who had already made up their minds a significant proportion of the questions were from people genuinely seeking information.

Did we change minds? Move the debate forward?

I’m not sure. The debate over GM Food has been going on for a long time and many people’s view are firmly entrenched. That said we think many visitors will have come away better informed about the wider issues associated with GM Foods.

The future?

The format worked. The technology worked. It would be good to run a zone on a topic where the battlelines are not already drawn. To find a topic where experts are still working out the issues, where they don’t yet know how best to articulate the issues, where feedback from interested parties and the public has some real value.

Posted on September 25, 2012

ModShane in Evaluation Reports, News | Comments Off

2011 Evaluation

In June 2011 we came to the end of a grant from the Wellcome Trust to run I’m a Scientist in 2010 and 2011. We’ve evaluated throughout the project and the final evaluation report is available to download below, as well as a shorter executive summary and summary sheet.

2010-2011 Evaluation Report – at 75 pages long it contains a lot information including the methodology, aims, key figures and data, findings and recommendations.

Executive Summary – this outlines the main findings from the 2010-2011 Evaluation Report and looks at whether the objectives set at the start of the project have been met.

Summary sheet of the event in 2010-2011 – this highlights some of the key numbers and quotes from evaluations, and reasons why the event works so well.

How do we evaluate?

Evaluation is absolutely key to understanding what students and scientists have got out of the event, and it helps us continuously improve the event. There are two types of evaluation that we’ve done on I’m a Scientist, right from the first pilot event back in 2008.

Formative: We have done formative evaluation throughout the project – asking students, scientists, teachers and stakeholders for their views and acting upon them.

Summative: This is the more formal type of evaluation where we ask participants for their views before the events starts, observe the event in action in the classroom and survey views after the event has finished.

One of the benefits of running an online event is that we collect an enormous amount of data about how students and scientists engage with the event. We know how many students asked questions, how many questions they asked, how much they participated in live chats. We can tell the scientists how many students they’ve reached, how many questions they’ve answered and whether they’ve changed the attitudes of those students.

We’re always looking for ways to improve our evaluation, but what’s the point of evaluation is no-one reads and learns from what you’ve done. We’re genuinely interested in hearing what measures you’d like to see. What is it you want to see in an evaluation? Send us an email with your suggestions.

Previous evaluation reports

You can find the evaluation report summary event here

You can find the pilot evaluation report here

You can find the complete evaluation report on the 2008 pilot event here (this is the full, 127 page long baby)

You can find a summary of the 2008 pilot event evaluation report here

Posted on March 4, 2012

modangela in Evaluation, Evaluation Reports, News | Comments Off

Sports Science Zone evaluation report

In March 2012 we ran a Sports Science Zone, funded by The Physiological Society. It was a big success.

It was the most popular zone requested by teachers
More than 2 sports scientists applied for each place
The students asked lots of questions on sports and the human body. Keywords that poopped up a lot include Body, Blood, Brain, Cardiovascular, Cells, Exercise, Heart, Human Body, Lung, Oxygen, Performance and Running. Discussions ranged from how extra organs would effect the human body to the best way to improve personal health.
Heart researcher Fiona Hatch won the zone and spent her prize money giving a sheep’s heart dissection lesson to a local school in Hull.

Read the full evaluation here.

As one sports scientist said “this has been a blast”.

Posted on February 13, 2012

in Evaluation Reports | Comments Off

Evaluation

Evaluation in 2011

In June 2011 we came to the end of a grant from the Wellcome Trust to run I’m a Scientist in 2010 and 2011. We’ve evaluated throughout the project and the final evaluation report is available to download below, as well as a shorter executive summary and summary sheet.

2010-2011 Evaluation Report – at 75 pages long it contains a lot information including the methodology, aims, key figures and data, findings and recommendations.

Executive Summary – this outlines the main findings from the 2010-2011 Evaluation Report and looks at whether the objectives set at the start of the project have been met.

Summary sheet of the event in 2010-2011 – this highlights some of the key numbers and quotes from evaluations, and reasons why the event works so well.

How do we evaluate?

Evaluation is absolutely key to understanding what students and scientists have got out of the event, and it helps us continuously improve the event. There are two types of evaluation that we’ve done on I’m a Scientist, right from the first pilot event back in 2008.

Formative: We have done formative evaluation throughout the project – asking students, scientists, teachers and stakeholders for their views and acting upon them.

Summative: This is the more formal type of evaluation where we ask participants for their views before the events starts, observe the event in action in the classroom and survey views after the event has finished.

One of the benefits of running an online event is that we collect an enormous amount of data about how students and scientists engage with the event. We know how many students asked questions, how many questions they asked, how much they participated in live chats. We can tell the scientists how many students they’ve reached, how many questions they’ve answered and whether they’ve changed the attitudes of those students.

We’re always looking for ways to improve our evaluation, but what’s the point of evaluation is no-one reads and learns from what you’ve done. We’re genuinely interested in hearing what measures you’d like to see. What is it you want to see in an evaluation? Send us an email with your suggestions.

Previous evaluation reports

You can find the evaluation report summary event here

You can find the pilot evaluation report here

You can find the complete evaluation report on the 2008 pilot event here (this is the full, 127 page long baby)

You can find a summary of the 2008 pilot event evaluation report here

Posted on August 8, 2011

in Evaluation, Evaluation Reports | Comments Off

So How Was IAS2010?

In our pilot event, we did nearly all of the evaluation in-house, to save money. We tried to be as objective as possible, but of course it’s much better practice to have an independent evaluator. This year, we’ve been able to do that, and we appointed the lovely Kate Pontin to evaluate the project for us.

We are publishing the evaluation report on 2010 today (and also presenting it at the Science Council). You can also read the team’s early thoughts on this year in our write up of our Science Online London session, or see our evaluation report on the pilot.

Download full I’m a Scientist Evaluation Report 2010 (pdf)

Summary of I’m a Scientist 2010 evaluation

This report summarises the findings of the first year of formal evaluation of I’m a Scientist, Get me out of Here! and thus acts as an interim report.

I’m a Scientist creates opportunities for scientists to answer questions from students via the internet, opening dialogue, while developing skills and understanding of the science process. It also gives students an idea of what it is like to be a scientist and the types of careers available.

Evaluation data was collected from scientists, teachers and pupils via surveys, and interviews. Observations of a number of sessions in school were also undertaken.

Data suggested that the IAS programme is very successful, providing a unique approach to the communication of science to students. It shows that:

Students gained a lot from the experience. Their feedback shows they thoroughly enjoyed the event, especially the live chat sessions. They developed key skills including focused questioning and gained confidence in scientific discussion. They found it interesting to discover more about what it is like to be a scientist and were surprised that scientists were not as stereotypical as they originally thought.
Teachers found that IAS supported their need to develop different approaches in the classroom and also to encourage pupils to think about science and who it is actually undertaken by. They found the resources very useful, in particular the debate kit and were also pleased with the support they received from the team during the running of the event.
Scientists also gained or developed skills in communicating with the public, explaining their work (often complex science) clearly to pupils aged 13 onwards. This in some also helped inspire and revive their enthusiasm for their own studies.

Minor improvements suggested are:

Helping teachers to prepare pupils especially in their development of clearly focused questions
Promote and signpost other debate lesson packs like the IVF debate, and to help conclude the event, with perhaps an assessment element.
Further initial guidance for scientists giving guidance on the time it might take, but also for those less familiar with students of this age some general information on KS3/4 and the diverse knowledge base.

Evaluation will in 2011 focus on finding out more about how to attract scientists and about the impact their involvement has on their organisations. Longitudinal studies will look at the impact of students in the longer term (for example in take up of science or exam results). It is also hoped to develop baseline and plenary activities to find out more about change in knowledge and attitude to science.

I’m a Scientist enabled students using an innovative approach to inspire and enthuse. It worked with pupils from high achieving classes but also those across the ability range and from a wide range of backgrounds including groups from BME families.

Key figures for 2010

Two events – March and June
7,459 questions asked
125 scientists were involved, in 25 zones
94,909 visits to the site
6397 students took part
648,563 page views

About the evaluator

Kate Pontin is a former geologist who has been working in museums and evaluation for over 20 years. She specialises in the experiences of young people in informal learning environments.

Posted on March 4, 2011

modangela in Evaluation, Evaluation Reports, News | Comments Off

So how did I’m a Scientist go in 2010?

It’s important, in anything, to check that you’re doing a good job. When we’re running I’m a Scientist, we think we’re doing a good job and that the event is fun and exciting, and that students (and scientists!) learn things. But we aren’t the most impartial people to check!

We’re spending lots of money on this project (given to us by a charity called the Wellcome Trust), and we’re also taking up lots of lesson time at hundreds of schools around the country. So it’s important we check that the event is doing what it’s supposed to.

So a researcher called Kate Pontin has been checking for us. She visited some schools when the event was happening (maybe she came to your school?). She watched lessons and talked to students. She also talked to some of the scientists and teachers. And she looked at the surveys some of you filled in at the end.

What she found out was:-
Facts and figures for 2010

Two events – March and June
7,459 questions asked
125 scientists were involved, in 25 zones
94,909 visits to the site
6397 students took part
648,563 page views

Students

Enjoyed taking part
Got better at asking good questions
Got more confident talking about science
Found it interesting to talk to real scientists and learn about real science
Learnt that scientists weren’t like the stereotypes

Teachers

Found it helped them try out new ways of teaching
Helped them teach How Science Works
Helped them teach students about science and scientists
Liked the classroom resources
Thought the I’m a Scientist team were helpful (*beams*)

Scientists

Got better at explaining their work
Some of them found it made them more into their work and remember why they liked it.

She suggested we should make some improvements for next year though

Add more classroom resources for teachers
Make it easier for teachers to use the ones we’ve got already – like make them more obvious
Explain more to scientists what level GCSE students are at, so they’ve got a better idea what to expect in some of the questions.

If you want to take part next year (and you aren’t already signed up) then use these links

Teachers – to find out more about taking part, and sign up

Scientists – to find out more about taking part, and sign up

Posted on November 11, 2010

in Evaluation Reports | Comments Off

The long awaited IAS evaluation report is published!

Here is is, all 126 glorious pages of it. The I’m a Scientist Evaluation Report [pdf 1mb].

If you’re interested in how the event went and what people got out of it then look at Section 1 (findings) and Section 3 (case studies). If you’re a fellow science communication professional, and want to know more about how we did things and learn from our experience, then look at Section 2 (development) and Appendix 3 (formative evaluation findings).

The key messages were:-

Students said: “i learnt loads and feel much more confident to put my hand up and ask questions and know that ok sometimes i will get it wrong”

Teachers said: “The pupils were looking forward to their science lessons, asking about them in the corridor etc.”

Scientists said: “It engaged the kids in a way I’ve never seen before.”

Every scientist and teacher surveyed said they would recommend it to a colleague.

Key Outcomes

Students realised scientists are real, interesting, fun people
Students were inspired and enthused
Developed debate and discussion and How Science Works (HSW) skills

Key reasons it worked

Giving some power to young people gives them a reason to engage and shows that they are trusted
The fact that it’s real – real scientists, real science, real prize money – makes it far more vivid
The intimacy of the medium makes it easier to break down barriers and make connections

The key lessons for others were

Teachers want to use more debate and discussion for the new GCSE but feel they and/or the students don’t know how to do it. So resources which help are welcome.
Many existing resources to support the new GCSE feel ‘rushed out’ and don’t always ‘get’ what How Science Works is.
A teacher panel is a really useful way to include teachers in development of your project.
A project blog is a great way to be transparent and keep everyone informed.

I hope this is useful to people.

Posted on December 12, 2008

in Evaluation, Evaluation Reports, IAS Event | Comments Off

What can you do?

I want to take part with my class ❯

I want to take part as a scientist ❯

I want to find out about funding I’m a Scientist ❯

What is I’m a Scientist?

I’m a Scientist, Get me out of here is an online activity where school students get to meet and interact with scientists.

It’s a competition between scientists, where the students are the judges.

Students challenge the scientists over intense, fast-paced, online live Chats. Then they Ask the scientists all the questions they want to, and Vote for their favourite scientist to win a prize of £500 to communicate their work with the public.

Follow @imascientist