Category Archives: Evaluation

Visits to IAS by technology type

As part of a review of what we need to do over the next 5 years with I’m a Scientist, we looked at how visitors accessed the site and how that has changed year by year over the last 4 years. We also looked at how visitors registered as students differed from All visitors. The Student visitors probably give a fair reflection of school IT capabilities. In the graphs the student data is represented by dashed lines. We only have student data from 2011 when we started tracking registered users separately.

IAS by device
It was no surprise to see that visits from mobile and tablet devices were increasing but having 23.6% of visits from mobile devices in 2013 was a shock. Delving deeper it is clear that it is mostly questions that mobile users visit. Tablets tell a different story. For student tablet users there is little difference in content visited when compared to student desktop users. And it is interesting to see that the levels of tablet use are the same for students as for all users at just under 8%.

We need to adapt our design for the question page so mobile users get a better experience.
We also need to be more aware of how easy it is for registered tablet user to interact with the site.

IAS by browser
Browser usage tells an interesting story. Unsurprisingly Internet Explorer is plunging. But at over 60% it is still the most popular browser in schools. More surprising was the drop in usage of Firefox. The rise in other browser is mostly due to mobile and tablet browsers with Android, Blackberry and Opera featuring.
IAS by IE
Finally the web developers enemy – old versions of IE. IE6 has just about disappeared. Yay. IE7 hangs on at 4%, but IE8 remains popular. The surprise for us in this graph was that All visits and Student Visits differed little in respect of IE version.

If you have any questions or would like more data. Please let us know in the comments or via email.

Posted on January 22, 2014 ModShane in Evaluation | Leave a comment

I’m a Scientist acts as a public engagement booster for scientists

Anecdotally, we’ve heard of how I’m a Scientist can be a good starting point for science communication activities. This was the case of Suzi Gage, Tom Crick or Suze Kundu, who took part in I’m a Scientist in June 2011 and are now putting a great emphasis on the communication side of their scientific careers or even fully devoting to it.

Twitter conversation about the impact of I'm a Scientist on Science Communication careers

Twitter conversation about the impact of I’m a Scientist on Science Communication careers

However, we were still curious to know to what extent we could extrapolate this to the wider community of scientists that have participated in the event during the last years. How could we know if I’m a Scientist had encouraged them to do more science outreach? Well, we decided to ask them.

We sent a survey to all the scientists that had participated in I’m a Scientist until 2012, leaving a gap of at least one year since they took part in the event. Approximately a quarter of the scientist responded to the survey, resulting in a more than decent sample of 113 scientists. The data collected in the survey show that there is a strong indication that I’m a Scientist is a real public engagement (PE) boost for scientists.

Summary of the survey's main findings

Summary of the survey’s main findings

I’m a Scientist is a good way to start doing public engagement in schools

If we look at public engagement at schools, we have recorded a significant increase, especially among those scientists that were particularly new to this form of outreach. Moreover, scientists who had never done school public engagement were encouraged to do more public engagement in general, going from none to an average of over 4 activities per year.  This was supported by some great comments gathered in the survey:

Having never done outreach with schools before, IAS gave me the chance to engage with a different audience than I would typically.” – Scientist

Scientists find the event flexible, open and inclusive

The online nature of the event was praised by different scientists that left comments in the survey:

I really liked how the online format broke down barriers and allowed the students to ask anything they wanted without having to stand up in a crowd.” – Scientist

Having all the activity online also gave me the flexibility to contribute more of my time, ad from a remote setting, compared to face-to-face school visits.” – Scientist

"Sifting the evidence" Suzy Gage blog hosted by The Guardian

“Sifting the evidence” Suzy Gage blog hosted by The Guardian

Some of the scientists had already contributed to the online scientific community with blog posts, podcasts or through social media. However, for a high proportion of them (68 out of the total 113) I’m a Scientist represented their first online public engagement event, which really pushed up their participation in general public engagement. In this case, their collaboration with public engagement activities went from zero to more than 3 general outreach events per year.

It was also interesting to find out that those who had already done lots of public engagement (4-15 activities per year) started doing more specific online outreach after participating in I’m a Scientist, at the expense of other forms of public engagement.

I’m a Scientist reveals itself as a great launchpad for budding science communicators

The majority of the scientists that filled in the survey (86%) had already taken part in some kind of public engagement activity (lectures, science festivals, interviews in traditional media, science policy making, etc.). It was a nice surprise to find out that scientists who had done very little public engagement (1-3 activities per year) increased their activity dramatically, going up to 5-8 outreach activities per year after the event. What is more, the greatest increase in overall public engagement activity (an increase of 130%) corresponded to the scientists who were just doing very little of it before the event.

Looking at the big picture, there is a general trend that I’m a Scientist enhances the participation of scientists in outreach events, except in the case of those who already did loads (more than 30 activities per year), where there is very little room for improvement. All in all, we are happy to confirm that I’m a Scientist represents a great launching platform for public engagement in science.

It remains the best public engagement event in which I have been involved.” – Scientist

 

Posted on January 8, 2014 modangela in Evaluation, Event News, News, Science Engagement, Scientists | Leave a comment

How does I’m a Scientist change students’ perceptions of science?

We’ve just run I’m a Scientist in Ireland and are curious to know how our event actually affects students’ attitudes towards science.

In order to do this, we included a short and compulsory pre-event survey in the form students used to register. We then asked students to fill in the exact same survey on their profile page after the event. When we matched the data from the two surveys, 92 students (7% of 1,247 students that participated in I’m a Scientist) had filled in both surveys. Importantly, data from the total number of students that filled in the pre-event survey very closely correlates with the pre-event data of this 92 student sample.

We were very happy to find out that students’ interest in science and science related careers is clearly increased after taking part in I’m a Scientist. This is what we have learnt:

The amount of students that say they love science doubled after taking part in I’m a Scientist

How does school make you feel about science?

How does school make you feel about science?

I’m a Scientist really got the students excited about science! Before taking part in the event, only 23% of students said they loved science, but this number increased up to 51% after the event. On the other hand, the number of students that don’t feel really excited about science or think it is boring decreased from 9% to 3%.

Participating in I’m a Scientist encourages students to choose a science subject in the next stage of their education

In the pre-event survey, 66% of the students were absolutely certain or very inclined to choose a science subject next year. However, this percentage raised up to 71% after participating in I’m a Scientist.

Thanks guys for talking to me really helped me make my decision for the leaving cert – sarahlawless, student

Students are keener on science related jobs after participating in I’m a Scientist

The majority of students that completed the surveys already thought that jobs involving science are at least fairly interesting before taking part in the event, but there was still room for improvement and the percentage of students that considered science related jobs very interesting saw a big increase from 36% to 62%.

Taking part in I’m a Scientist increases the likelihood of students looking for a job that uses their science skills

How likely are you to look for a job that uses your science knowledge?

How likely are you to look for a job that uses your science knowledge?

Before taking part in I’m a Scientist, a big portion of the students (37%) couldn’t decide whether they would try to look for a job that uses their science skills and only 10% said they were sure that they would look for this type of job. However, the event seemed to be the boost that students needed to be more confident about looking for a science related job. After I’m a Scientist, most of the students (68%) said that they would certainly or very probably look for a job that uses their science knowledge.

Honoured to have taken part. The future of science is in very good hands with you guys! – scientist

In addition to this, students left several comments that stated clearly how they were enjoying and learning at the same time throughout the event. They liked that the event was so interactive and that they had an active part at every step: asking, commenting, chatting and voting.

In the future, we would like to use this same strategy to measure the impact of other events or activities. We are also very interested in analysing the gender and year course differences that could be found in the pre-event data.

Posted on December 19, 2013 modangela in Evaluation, Event News, IAS Event, News, Science Education, Science Engagement, Teachers | 1 Comment

IAS Live: Antibiotic Awareness Day

Yesterday afternoon, on European Antibiotic Awareness Day, we joined the Royal Society of Chemistry (RSC) at Burlington House in London, Piccadilly for a foray into antimicrobial research.

CAPTION

I’m a Scientist: Live at Burlington House.

We took five scientists — three of whom were I’m a Scientist alumni, having taken part in previous online events — whose research looks at the uses of antibiotics and antimicrobials, put them in a room in-front of around 100 sixth-formers, and had them answer questions on everything from Typhoid Mary, to zombie apocalypses.

Our scientists were Rob Shorten, Jess Bean, Mark Roberts, Clare Taylor, and Emma Newton.

CAPTION

Jess Bean and Mark Roberts go for the BAFTA puppeteering award with plush microbes.

Helen Arney, our MC for the afternoon kicked off by introducing the scientists before launching into a round of Science Fact or Fiction. The facts covered everything from bacteria flavouring cheese and yogurt, to our being only 10% human, with most of the cells in the body (by number) actually being bacteria. Clare introduced the notion that around 3% of the population can carry Salmonella their whole lives without ever becoming ill; setting up a whole round of questions on historical poisoning and Typhoid Mary.

We then moved on to the questions round: Emma presented a world first, showing us her “electronic nose”, a prototype device for sniffing out different bacteria based on the chemicals they produce. Using plush microbes, Jess and Mark acted out an Academy Award worthy demonstration of a bacteriophage’s attack on an E coli cell. Rob took us through the purposes of a faecal transplant, Mark explained how some bacteria can follow the earth’s magnetic field, while Clare took the audience award to the best piece of advice for surviving a microbial zombie outbreak: “there’s always a percentage of the population with a resistance; seek them out and make a cure.”

In the end there could only be one winner. After giving the scientists a last plea for votes the votes were counted yielding a surprisingly close spread, with Clare standing above the others to take hold the coveted I’m a Scientist: Live trophy and mug.

Let’s evaluate

CAPTION

Standard end of event competitors’ photo, accompanied by MC, Helen Arney.

The room was filled, with around 100 students in their seats, votes being counted using the electronic voting system were all around the 90 mark. The audience was made up predominantly of school students, with about 12% being teachers.

The use of electronic voting pads gave us the chance to do some on-the-spot evaluation of how the audience found the event. A majority of the audience who provided this feedback said they had learnt something with 49% saying they had “learnt a lot”, and 37% learning “a bit”; only 8 respondents (14%) said they had not learnt anything.

To the question, “Has the show made you want to find out more about antimicrobials?”, 50% of those who replied “got the bug”, while 18% had had enough. Owing to a slight technical fault we weren’t able to count the replies to, “Did you enjoy the show?”, what we did get were a whole lot of positive comments from students and teachers.

I'm a scientist live - RSC - Did you learn anything?I'm a scientist live - RSC - Has the event made you want to learn more about antimicrobials?
The audience seemed to really engage with the scientists, asking a good mix of in-depth scientific questions, as well as some more sillier ones. This was helped in no small part by Helen’s fabulous job setting the tone, demonstrating that there could be no such thing as a bad question; and putting the audience and the scientists at-ease.

Really informative; loved the bit when the scientists talked about their specialties; really interactive – the pads were great; great venue and they liked the refreshments (felt very adult!)

Many students stayed behind to ask more questions to the scientists once the event was over, and working by the buzz in the room, and the comments from teachers and students, it seems like the afternoon was a resounding success.

What have we learnt?

IAS-Live

  • Scientists bringing props is a good thing.
  • Being nice and altruistic can get you lots of votes, or certainly lots of applause.
  • It’s useful to run through the technical aspects of the event maybe a couple more times than might feel necessary; can’t be too prepared.
  • Students appreciate when a refreshment buffet includes cake.
Posted on November 20, 2013 Moderator - Josh in Evaluation, I'm a Scientist - Live | Leave a comment

IAS Live: Drugs, Bugs and Infections

Last night five scientists researching different aspects of antimicrobials took to the floor at the FIS Conference at the ICC in Birmingham to take part in I’m a Scientist Live: Drugs, Bugs and Infections.

Photo courtesy of @biologyatkinge1

Photo courtesy of @biologyatkinge1

Our five were:

Mark Webber

Richard Bax

Amy Wedley

Anne-Marie Krachler

Peter Hawkey

AMK & LP

Photo courtesy of @PharmaMix

Our MC for the evening, Simon Watt, kicked off by spending 10 minutes interviewing Laura Piddock, Chair of Public Engagement at BSAC about the current situation in research. The scientists introduced themselves and gave their Science Facts and Fictions before the 100-strong audience launched with their questions.

30 mins later with time running out, Simon brought the event to a climax by asking the scientists to spend 30seconds telling the audience why they should get the votes and the winner’s trophy. The vote that followed confirmed that Anne Marie’s promotion of the IMI’s Summer School for Year 12 students was enough to edge her into the lead and win the vote.


Some numbers:

IASLIVE FIS Who are you

181 tickets ordered.

105 attendees on the evening (105 voting handsets were used).

Over half the audience were students evenly split between schools and undergraduates. This means a lot of the students who asked for tickets failed to show. We had a capacity of 150 for the evening with a maximum limit of 200 for the room. Whilst we expected some dropout we are surprised by the level of no-shows. We will need to over-book more heavily in future.

Voting was close. Only one vote separated Anne-Marie from the scientist who was second-placed. We discussed what to do in case of a tie beforehand, but were clear it wouldn’t end up as a draw. We need to take those plans more seriously in future.

Having the electronic voting cards allows us to ask some immediate evaluation questions. A healthy majority enjoyed the show and learnt from our panellists. A slightly smaller majority have caught the antimicrobial bug.

IASLIVE FIS MoreIASLIVE FIS Learn


What else did we learn/have confirmed from last night?

IASLIVE FIS Enjoy

  • Allowing a 30 minute turnaround between a lecture and IASLive with only one sound engineer is a little optimistic.
  • We should brief participants that they will be using a lapel mic and that they therefore need a lapel.
  • Some sort of stage helps if the audience are on the same level
  • Enthusiasm and energy will give rise to a decent discussion whatever the subject.
Posted on November 20, 2013 ModShane in Evaluation, I'm a Scientist - Live | Leave a comment

What criteria do students judge scientists on?

When students take part in I’m a Scientist they get to vote for their favourite scientist to win £500 to spend on communicating more science. It gives them ownership of the project and they decide who gets, in effect, a small grant for public engagement. But what are students’ votes based on?

This is how 855 students (or groups of students) have ranked certain criteria from most to least important when considering how to vote for scientists, since January 2012. The results come from a Drag & Drop ranking activity in the first lesson plan “You’re the Judges” that teachers run to introduce their students to I’m a Scientist.

Voting ranking criteria

The top ranked criteria are all linked to health or the scientific method. This is reflected in the number of questions students ask scientists about saving lives and animal testing. Reassuringly, the more superficial criteria lurk at the bottom of the list.

Other criteria that students suggested include:

“Abides to Scientists ethics, and morals”

“A person that thinks ‘outside the box’. Confident person. Optimistic person”

“Enthusiasm”

“How much they believe in and care about their work”

“The speed that they answer my questions. Also if they turn up for a chat”

“How great their beards are”

“If the scientist works at weekends (as well as during the week)”

And finally, the crux of all scientific research:

“Whether their tests are fair or not”

One teacher commented that, “the class disagreed on the ‘good-looking’ criteria because some feel that people are judged by first impressions and looks are included in that”.

Teachers, how do these criteria compare to how your students judge the scientists? Did students generally agree on the rankings or did certain criteria provoke more discussions than others?

Posted on November 8, 2013 in Evaluation, IAS Event, News, Teachers | 4 Comments

Using teacher feedback to plan future themed zones

A question I sometimes get asked when I tell people about I’m a Scientist, is how do we choose which zones to run? In short: we ask teachers what they’d like.

In July we emailed all the teachers signed up for I’m a Scientist to ask them what zones they’d like to see in our next three events: November 2013, March 2014 and June 2014.

Just over 40 teachers filled in each survey by ticking as many of the 40 zone choices as they’d like (some are zones we’ve run before, others are new).

Which zones came out top?

20 zones

The graph shows the top 20 zones, by the number of teacher requests across the 3 surveys for November, March and June. Forensic Science and Genes zones came out top, closely followed by Stem Cells, Health, Organs and Sport Science. Most of the top requested zones are biomedical; this might be because there are more Biology teachers on our list, due to the majority of funding coming from the Wellcome Trust since 2008.

When to run these zones?

What’s more interesting is the variation when teachers want to see certain zones, reflecting when topics are taught in the school year.

  1. Stem Cells, Drug Development, IVF and Subatomic zones were more requested in November than March or June.
  2. Earth, Chemicals and Technology zones were less requested in March than November and June.
  3. Communication and Human Limits zones were more popular in June than November or March, while Blood and Reproduction zones were less popular.

So what…?

We’ve used this feedback to choose some of the zones we’ll run over the next year in advance. This will help us plan teacher and scientist recruitment throughout the year. STEM contract holders have also told us it’s useful to know dates and zone themes in advance.

And sad news for any linguists out there, I don’t think we’ll be running a Language Zone any time soon. With only four requests there just isn’t the demand.

A taste of the themed zones running over the next year is at imascientist.org.uk/2013/10/im-a-scientist-zones-november-2013-june-2014. We’ll add more themed zones from other funders to the mix nearer to the events.

Posted on October 30, 2013 in Evaluation, IAS Event, News | Leave a comment

We’ve published all our zone reports from June

We’ve published all our individual zone reports from the June 2013 event. If you delve further back into the archives, you’ll find them for the zones we ran in March 2013 & November 2012 too. All the reports are tagged (funnily enough) under “Evaluation Reports“.

Each report sums up the activity in that zone. They provide some information for the scientists and teachers in that zone about how much the students and scientists interacted.

How many questions were asked and answered? What were the popular topics asked by students? What examples of great engagement stood out? What did students, scientists & teachers say about taking part? How does the winner plan to spend their £500 on communicating more science?

We email out each report to all the scientists and teachers in that zone. Scientists tell us the reports really help justify their participation to supervisors and colleagues. The detail tells them far more than just receiving a participation certificate in the post. The ‘loads’ of questions they answered turns out to be in the hundreds. The live chats went in a flash thanks to around 300 students bombarding them with questions over the 2 weeks. And the students really did ask about their research – volcanoes, bacteria & cancer for the zone below!

Technetium wordle

Take a look. If you’ve got any comments, such as what other info might be useful for scientists and teachers, we’d love to hear them.

Posted on July 17, 2013 in Evaluation, News | Leave a comment

Who are you Twitter?

As we prep for our Science Communication Conference session on how we evaluate I’m a Scientist, the occasional sidetrack leads to interesting stuff.

A while back we used Tony Hirst‘s scripts and know how to download our twitter followers network. We’ve used Gephi to map out the clusters within our network which helps us work out who is following us and how diverse our network is. We were delighted to see an Australia cluster thanks to our friends in Adelaide. More about that later.

Another useful benefit of downloading your followers is the ability to search their bios to find specific followers who are interested in certain subjects. This evening we identified 83 followers with “food” in their bio. We’ll be contacting some of them (excluding those who are just “foodies”) to tell them about our new debate kit on Food Security.

Whilst I was there I thought I’d create a little wordle with big word in it!

100 word Wordle of Bio of followers of @imascientist

100 word Wordle of Bio of followers of @imascientist

To create this wordle I exported the Gephi table into Excel and converted the bios into lowercase and change sciences into science. More cleaning of similar words would create a better wordle but time is always short. Then I pasted the words into Wordle, set a maximum of 100 words, and removed some of the irrelevant words such as twitter and tweet and like. Set some Custom Colors and tweak the layout until it worked and you can see the result. Enjoy.

Posted on April 18, 2013 modshane in Evaluation, Science Engagement | 1 Comment
Selection of IAS Postcards

IAS Outreach by STEMNet Area

We’ve selected the scientists and schools (all will be revealed next week) and we’ve looked at where they are across the country. We want to work closely with STEMNet contract holders to promote our events to teachers, scientists and engineers, so we’ve calculated how many schools, classes and scientists are in each STEMNet contract area. From that we’ve calculated the estimated number of interactions (or Live Chats as we call them) we expect per area. On average each class has one live chat and each scientist attends seven.

The best news was that there are only 3 areas where there is no activity. We are doing science engagement and enrichment across the country.

But where do we have most activity?

Most classes signed up:

  1. Bristol, Bath and Somerset – 25
  2. Lancashire – 19
  3. Surrey, Central and East Berkshire & West Yorkshire – 18

Special Kudos to Liz Lister in Bristol, Bath and Somerset – over 13% of secondary schools in the area are signed up.

Most scientists:

  1. London North – 6
  2. Bristol, Bath and Somerset – 4
  3. Greater Manchester – 4
  4. Oxfordshire, Buckinghamshire and West Berkshire – 4
  5. Surrey, Central and East Berkshire – 4

We are very keen to work with all STEMNet Contract Holders to get their STEM Ambassadors taking part in the event and getting schools in their area enjoying the benefits of reaching scientists from across the country and world.

We have a promotional pack that we can send out containing a flyer, video and sets of postcards for both teachers and scientists.

Selection of IAS Postcards

IAS Postcards from the Promotional Pack

Even better, we have a large network of teachers and scientists who may be willing to give a 5 minute presentation at your networking events to discuss how I’m a Scientist helped them enthuse students about science and engineering.

Please contact us if you would like help in getting more STEM Ambassadors and schools involved – rosie@gallomanor.com or 01225 326892.

Posted on February 7, 2013 modshane in Evaluation, Event News | Leave a comment

What themed zones do teachers want?

For I’m a Scientist to work we need to run zones that teachers and students want. So when we’re deciding on zones to run in the next event we ask teachers to tell us what they want.

Teachers voted on a longlist of zones we drew up – they could select as many as they’d be interested in taking part in. We also asked them for suggestions for other zones.

Over 100 teachers told us what zones they’d like to see. The results are below.

Posted on December 12, 2012 in Evaluation, IAS Event, News, Science Education | Leave a comment

IDACI Index Profile

It it important to know who we are reaching with our events. Funders are keen to know and we want to know that we aren’t just reaching the usual suspects. We’ve recently done some geographic analysis, which we’ll share soon.

But general location is only part of the story.

Nice schools in nice areas tend to do plenty of science engagement and send lots of students to university. Therefore it is important to see if we are reaching those schools who don’t normally participate; those hard to reach schools.

But how do you measure that?

Does the deprivation of the area in which the school is located provide a good proxy? Some studies would suggest so: “The Characteristics of Low Attaining Pupils” DfES Statistics Bulleting 02/2005 June 2005

We plotted the IDACI Score of the schools who participated in IAS and compared the profile (based on deciles) against that for all schools. You can match postcodes to IDACI thanks to a nifty tool from the Department for Education
The problem is that most schools are in an area with a score of between 0.1 and 0.3. It isn’t granular enough to see much difference. BTW the lower the score the less deprived the area. I suspect this is because most areas also fall into that range.

So we then looked at the ranking. IDACI area is ranked between 1 (most deprived) and c. 33,000 (least deprived). We then grouped into groups of 3,000 and plotted a histogram. This time the All Schools distribution is much more even allowing us to spot differences between IAS and the national distribution.

And so what?

In the most deprived quarter of areas we follow the national profile, but midway we slip behind and find ourselves with more schools than expected in the least deprived areas. It is no surprise. It is our expectation that schools in the least deprived areas will be more likely to find time to seek out and run more engagement and enrichment projects.

But,

I really want to know how our profile compares to other science engagement projects. Are we all hitting the same schools? Are there differences that we can build upon to ensure the widest reach?

Please use the comments to give us any feedback or suggestions of where to go with this.

Posted on November 23, 2012 modshane in Evaluation | Leave a comment

SpotOn London 2012 – Background

As part of ScienceOnline (or SpotOnLondon2012) Karen Bultitude and I have arranged a session called: Can we work together to better evaluate online engagement? It’s taking place in the Steel Room on Monday 12th November at 2.30pm.

We wanted to provide some context to the session. Here goes:

Since you’re interested in ScienceOnline it’s fairly likely that you think online science engagement is important and going to get more important. Me too. I run a few online science engagement projects, http://imascientist.org.uk is the most prominent. It’s pretty important to me to know how well the project is performing. We need to know in order to improve it, we need to be able to tell our funders and we feel obliged to be able to demonstrate to all stakeholders that the project is effective.

So we evaluate our projects using a wide range of methods: surveys, interviews, personal meaning mapping and we also try to analyse the vast quantities of data that we capture as a matter of course from the website and Google Analytics. We use that information to work out how we’re doing.

All well and good except that without anything to compare against it is difficult to know how we are doing. A starting point when wanting to compare against sector standards is knowing what to measure. It is surprisingly difficult to find much information about how online engagement properties are performing. We try to publish all our evaluation. Google: evaluation I’m a Scientist. The first three results are our main evaluation reports and more informal evaluations of two spin-offs. Try it for another online engagement project. Google: evaluation [INSERT PROJECT NAME] – very little seems to be published.

The Cultural Sector (including The Science Musuem) recognised this problem in 2009. They came together to produce the “Let’s Get Real: How to evaluate Online Success” report in order to better understand their digital activities and to help them plan future activities more effectively in more financially constrained times.

The report brought together 17 museums, galleries and venues. They shared statistics and methodologies. They agreed a common configuration for Google Analytics for the purpose of reporting to funders. They shared good practice.

There is plenty of interest in how we measure our online activity. This session is the 3rd on the subject in this conference alone. Our hope is that we can work together to better measure our online success.

There will be benefits to better measurement. We’ll be able to improve what we do. We can more more efficient, more effective. We’ll spend less time wondering what, out of the mountains of data, is important. We’ll also be able to demonstrate to funders in a coherent and consistent manner that online engagement is effective. And that is something that will benefit all of us who believe in online engagement.

Posted on November 9, 2012 ModShane in Evaluation | Leave a comment

We've cut our teacher survey from 41 to 18 questions

This morning I cut down the survey we ask teachers to fill out after taking part in I’m a Scientist. The old survey was a hefty 41 questions long, and the new one has just 18 questions.

By looking at teachers’ previous responses to the 41 questions we were able to identify the questions which worked, and the questions which didn’t. Using this information we’ve made the survey simpler.

It’s also more quantitative. Funding from the Wellcome Trust for the next 3 years allows us to improve the way we evaluate I’m a Scientist, and move to evaluating outcomes through more quantitative measurements.

My next task is to do the same for the scientist and student surveys, and cut them down from 28 and 25 questions.

Posted on November 23, 2011 in Evaluation, IAS Event, News | Leave a comment

Evaluating the Impacts of engagement

How can we evaluate the impact on students taking part in I’m a Scientist? Can we measure if they’re more likely to take a STEM subject at A Level? If they’re more likely to study science at University? How should we use the large amounts of data generated by online projects? How can we share our evaluation in a more useful way?

These are just some of the questions we’re trying to answer about evaluating I’m a Scientist and other Gallomanor run projects. Judging from the first in a series of seminars looking at Evaluating Impacts of Public Engagement and Non-Formal Learning, last Friday 4th November, others are thinking along the same lines.

The Core Issues & Debates seminar kicked off the series at the Dana Centre in London, and bought together a range of researchers, evaluators and learning and communication practitioners. Future seminars focus on areas such as how to reach new audiences, evaluating online engagement and using qualitative evaluation methods.

The 7 speakers approached evaluating impacts from different views – funding, strategy, science festivals, academic, and museums/science centres. There were some key themes that emerged during each of the 20 minute talks and the resultant Q+A sessions. (It would have been useful to have a bit more time for Q+A discussion after each speaker, as the allocated 10 minutes were quickly eaten into.)

  1. Evaluation needs to be shared with others so all projects are ‘learning projects’. The British Science Association’s Collective Memory is a good place to start. It’s worth constantly thinking about how to improve evaluation during a project, such as changing evaluation questions so they return more useful responses.
  2. Evaluation is very important right from the grant application stage at the start of a project, but shouldn’t be done for the sake of it, or just because funders ask for it.
  3. There are still lots of questions unanswered about how to evaluate and measure the impacts of an engagement project. Is it really possible to measure if students are more engaged with or interested about science as a direct result of one activity? Is it enough to accept your activity is one of many factors that may have influenced a change seen? These will hopefully be explored further, and maybe even answered, in future seminars in the series.
  4. Negativity can be hard to capture in evaluation. Evaluation studies can therefore be designed to try and capture negativity, such as framing questions to encourage participants to think not just about the positives of the event.
  5. Bad evaluation that draws inaccurate or invalid conclusions from data can be more damaging than no evaluation.

Overall it was a useful introduction and summary of how impacts are being evaluated. Armed with my 7 pages of dense notes scribbled during the seminar we’re now working out how to put some of these ideas into practice with I’m a Scientist. This will likely spark another post in due course.

Click on the RSS symbol at the top left of the page to subscribe to the blog, or register for email updates.

Posted on November 9, 2011 in Evaluation, IAS Event, News, Science Engagement | Leave a comment

Read about our session at Science Online conference

We’d never been to the Science Online conference. And we’d never given a presentation about I’m a Scientist, Get me out of Here! But when the organisers asked if we’d like to do a session on the event we jumped at the chance.

We then closed the office for the whole of August, so we had to write and organise our highly interactive, possibly risky session in 2 days when we got back last Wednesday. At the same time as do all our catching up from the holidays.

Last week was an interesting week…

So here is a summary of our session (last Fri, 3rd Sept) and what we, and others, said.

Our presentation

Photo of Sophia speaking, in front of prezi presentation and a twitterfall about the session

Sophia presenting at Science Online coference London

At Gallomanor we like to walk the walk, as well as talk the talk. So we wanted to make the session as interactive and audience-led as possible. After a short (2 min) intro, we showed the audience the 12 topics we’d prepared and asked which 6 they wanted to here about.

Here’s the prezi presentation, with a slide on each topic. I’ll summarize below what we were going to say on each – including the ones we didn’t cover on the day.

IAS Solo on Prezi

1. Film – we showed our ‘Intro for the classroom’ film.

2. Our Philosophy

The audience didn’t vote to hear about this, which was a shame as I think it’s really important in explaining why we do what we do, and how. Fools, fools!;-) Maybe we should have called it something more exciting sounding…

Anyway, key points would have been:-

  • We reverse normal power structures (kids in our event are given some decision-making power and the chance to ask what THEY want to ask). This engages pupils who are normally turned off. See more on my thinking here in this piece on the Secrets of Engaging Teens.
  • Making it fun and game-like is not a gimmick and doesn’t make something not educational. It makes people pay attention and be interested! (e.g. this research)
  • It’s not just about getting kids to study science. Not all teenagers will grow up to be scientists, but they will all grow up to be people, and need to have a relationship with science. More on this point in this article for Wellcome Trust blog.
  • Scientists have a lot to gain from engagement too – they aren’t just doing everyone a favour. They can be energised, challenged and made to think by the huge variety, and inventiveness, of students’ questions.

“a 4.00am Eureka! moment solved a problem that’s been bugging me for the best part of a year. It came from a seed of an idea planted in my head by a simple question from a 13/14 year old, absolutely bloody marvellous! Next day I was skipping into work like a refugee Munchkin from the Wizard of Oz, bleary eyed but elated.”

3. History of event

Shane thought of the idea for our sister event I’m a Councillor, Get me out of Here! while drinking Guinness. It’s been running since 2002, helping councils and local councillors to engage with young people in their area.

We came up with the idea of doing a science version in 2007 and ran the first, pilot event in 2008.

4. Funding

The pilot was funded with a People Award from the Wellcome Trust. We then successfully applied for a follow up grant from Wellcome’s Society Award scheme.

We love Wellcome and they are fab. We approached lots of people – people like BIS and Dept of Education who you’d think would be the right people to approach, and we got nowhere. All roads led back to Wellcome.

If you’d like to apply for science engagement money from Wellcome, our advice would be:

  • Make sure you understand what they value and what they are looking for, and that your project fits in with that.
  • Contact them early and talk to them.
  • Make sure you have really thought it through and explained your plans (not just vaguely). They will fund imaginative, even risky stuff, but they want to know you aren’t going to piss the money up the wall.
  • Realise the importance of formative (and summative) evaluation.

5. Site detail

Shane was just going to run through what’s on the site and how it works. You can prob just have a look for yourself.

6. Site build

It was in WordPress, cos we like and support open source, etc. And also cos WordPress is great. Apparently the site is doing things with WordPress that no one has done before. This is of course very exciting.

Developed by total legend Mike Little. Here’s Mike’s presentation about the project at Wordcamp 2010.

7. Scientist Recruitment

We pursued as many routes as possible to recruit scientists, including contacting learned societies, universities, research institutes and companies. Personally, I also bore people to death at parties and force my card on them if they are scientists or know any.

Because we’d done formative evaluation (i.e. talking to possible end users) as part of the development, we were able to focus communication and explain what the benefits would be to the scientists.

Once we’d run the pilot, word of mouth was one of our most powerful tools, as the scientists who took part really loved it. We also had evaluation evidence to back up our claims of the benefits.

To select the scientists (as we are now oversubscribed) we involved our end users. For the last event we took each scientist’s one-sentence description of their work and put it up blind (i.e. just their words, no information on age, gender, ethnicity, organisational affiliation) on a website where students and teachers rated each description. We, and a representative of the Wellcome Trust, also rated each scientist and we combined the scores.

We still want to find new ways of recruiting scientists, and particularly reaching the scientists we aren’t reaching at the moment. If you can help, let us know! And if you want to get involved, sign up here.

8. Teacher recruitment

Timeline for debate kit sign ups, kit 1

Again, used many routes. As you can see from this graph of sign ups for debate kits, some of the most powerful methods are direct mail, the Planet Science e-newsletter, and posting on the TES message board.

Also, for the big event in June, we got a significant number of sign ups from the STEMNET newsletter and a few from many other sources. We think it’s worth casting the net widely.

Again, word of mouth incredibly powerful. Especially because we recruited a ‘teacher panel’ of teachers to help us develop the project, before the pilot, and they became very invested in the project and really helped us a lot. This emphasis on consultation with teachers also meant we really were providing something they wanted, in a way that was useful to them.

Cross-marketing from our debate kits project was also incredibly useful. It enabled teachers to get a sense of where we are coming from, and the quality of what we provide, before making the commitment of signing up to a two week event.

We are oversubscribed with teachers too, and here part of our selection procedure is to deliberately pick as varied schools as possible (geographically, type of school). This has included Special Schools, Pupil Referral Units and Bristol Hospital Education Service. And also schools in Shetland, Northern Ireland and Oman.

We also try to get a representative sample results-wise. By which I mean, high-performing grammar schools are overly represented in the schools who apply, we try to even things out by picking more of the less academically achieving schools. We think it makes a bigger difference to those kids.

As with scientist recruitment, this is still a work in progress. If you can help us get the word out to teachers (taking part is FREE to schools), please do! Or if you are a teacher who would like to get involved, sign up here!

9. Evaluation strategy

Formative evaluation has been absolutely key. We started talking to scientists and science teachers before we did anything else. Asking them about what they wanted, what would work for them, what motivates them.

For teachers, we recruited a teacher panel of people interested in the project, who could give us instant feedback via email on everything from lesson plan ideas to terminology on the site. This also meant that they were invested in the project, when the pilot came around, and understood it and what they could do with it.

We also included young people, for example testing possible designs on them. And we made several school visits to observe science lessons.

We had a limited budget for the pilot, so much evaluation had to be done in house. But we set aside money to appoint an external evaluation consultant, Yvonne Harris, to spend a few days on the project. She advised us throughout, and also conducted some independent interviews with participants at the end, and audited our report. This was absolutely invaluable as she could check things like questionnaires and methodology as we went along, and suggest solutions we would never have thought of, and bring an independent perspective to the whole thing.

It also helped that we had grown the project organically, as we had developed over the years questionnaires that worked, and found (and corrected) many useability issues as we went along.

You can read our evaluation report on the pilot here.

We now have a much bigger budget for the roll out of the project, and we have devoted far more of that to external evaluation. Kate Pontin is now our external evaluator, and she has been invaluable in helping us think with clarity about what we need to find out and how we can do it. She has also been able to do far more schools observation than we can, as we are busy running the event while it’s on! This has been extremely useful.

Kate’s interim report will be unveiled at a special event at the Wellcome Trust on 20th October. This will be part of a special ‘Beyond Blogging’ event, curated by us. Tickets will be very limited, but do get in touch if you’re interested in attending.

10 Evaluation findings

Shane outlined some topline figures so far from the 2010 events.

TopLine Web stats for IAS June 2010

  • 4,667 students
  • 100 scientists
  • 171 teachers
  • 6,580 questions
  • 3,085 comments
  • 4,744 votes

More in-depth evaluation results will be published on 20th October, as above.

In the meantime, you can read our full evaluation report on the pilot here.

A summary of the pilot evaluation report here.

Or our short evaluation report on the 2009 event here.

11. Summary of the strengths and weaknesses of this format

Strengths

  • Power reversal truly engages and empowers.
  • Online gives access to scientists (for schools) and students (for scientists), without having to go anywhere.
  • Doing it online also creates intimacy (makes it easier to ask real questions and break down barriers), compared to a scientist in person giving a talk.
  • It also ‘levels the playing field’ – quieter and less confident students participate more.
  • You’re reaching all the students in the class – not just the very keen, as you find in science clubs, etc.
  • A conversation develops over time – over two weeks, students can read about the scientists, go away and find out more, ask questions, think about the answers, ask more questions…
  • For teachers, the preparation work is done for them and they can concentrate on facilitating learning.
  • For scientists, it saves time – no travelling to a school, they log in from their desk. Every moment spent participating is spent in engagement.
  • Scientists also find the range and energy of the questions rejuvenating, thought-provoking and inspiring.
  • And scientists get into the competitive aspect and have fun.

Weaknesses

  • Some scientists (and some teachers) don’t like the informality of the project, although we think it has real value in making connections and breaking down barriers.
  • We’ve disguised the learning and made it fun, so some teachers/scientists/students don’t see that it’s there and think it ‘won’t help them pass exams’. (God help us if that is the only thing some people think education is about).
  • Schools IT (sigh!). Often school firewalls are over-enthusiastic and we do have some problems with schools blocking the site.
  • It can be hard work for the scientists. Some had ~700 questions to answer!
  • It’s expensive to run (although not compared to many other projects).

12. The Question Game

Our question comedy improv game! OK, so not very comedy, and not actually improv, but kind of a game, inspired by Whose Line is it Anyway? Audience members shout out a word, any word, and we search the site to see if there are questions (or answers) containing that word. Intended to give an insight into the enormous range of questions.

Here’s some results

Search dinosaur

Search space

Search evidence

If you want to play the question game for yourself (be warned, browsing the site can be addictive!), just go to the main page and type in the search box near the top.

Live chat

After the presentation, we wanted to give the audience a feel for what live chats are like, and why they are so popular with scientists, students and teachers. So we had a live online chat, with the audience taking the place of students. We gave out log in details and everyone in the audience who had a laptop (quite a few, it being Science Online) could log in and take part. We also showed the chat on the projector.

Some of the key benefits of live chats:-

  • Access to people who couldn’t be there otherwise: We had a scientist in Michigan, one in Sydney, a teacher in Shetland, and a student who was in school. All of them had taken part in the event and answered questions from the audience about what it was like.
  • Immediate and friendly
  • Fun
  • Newer comments appear at the top, so you need to read upwards.
  • Discussion isn’t threaded – we’ve found that breaks up the chat too much and stops it being a communal experience.
  • Chats can be difficult to follow at first, but you get used to it quickly. Students are often quite familiar with chatrooms and don’t find it’s a problem.
  • There are two chatrooms side by side, one for students, one for the scientists. (In this chat that means one for audience and one for our participants). This makes it more difficult to understand an archived chat, however, we’ve found from experience that if the scientists and students are all in the same box, the scientists’ replies get lost in an avalanche of comments from students, so this works better.

Archive of tweets relating just to our session, hashtag #iassolo.

Posted on September 7, 2010 modshane in Evaluation, General, IAS Event | 4 Comments

Students! Tell us what you thought about I’m a Scientist, and win a prize.

We're listening! Anatomy of the outer ear. Credit: Medical Art Service, Munich / Wellcome Images

This is a last reminder: We really want to hear from you!

  • What do you think, did you like I’m a Scientist?
  • What did you like/dislike about it?
  • Was it better or worse than normal lessons?

You should fill in our survey and tell us what you think. It helps us make it better for next time. Also you’ll go into a prize draw for £20 WH Smiths vouchers.

Survey is here http://www.surveygizmo.com/s/307117/i-m-a-scientist-june-2010-tell-us-what-you-think-

Thanks for your help!

Posted on July 19, 2010 modemily in Evaluation, Event News | Leave a comment

How DOES magnetism work?

He doesn't know either: Gilbert demonstrating the magnet before Queen Elizabeth /Wellcome Images

Last year I saw A C Grayling talk on happiness and it’s importance. He said that he bans his students from using the word ‘happy’, that it’s a lazy portmanteau word. He thinks that if you are forced to choose a different word – hopeful, exhilarated, content – you’ll think about what you really mean far more clearly.

I think the same can sometimes be true of jargon. Scientists taking part in I’m a Scientist have told us before that explaining yourself without using jargon is hard work, but unexpectedly rewarding. It means you have to think through what you mean and it exposes your mental shortcuts.

We saw a great example of this in Imaging Zone. A fairly innocuous-seeming question (Why do magnets attract and repel?) pretty much lead to the scientists realising that they don’t actually know how magnets worked. Not really. Not when they tried to actually explain it to other people, without using jargon.

Now these scientists include a man who spends many of his days working with an fMRI scanner, containing a magnet so powerful that you have to remove any ferrous object from your person before entering the room. But still, magnetism turned out to be one of those things that he learned about years ago, and sort of assumed he understood.

I’m expecting that some of you have the same feeling I did when reading that Q+A – ‘OMG, I don’t REALLY understand how magnets work either. How did I not notice that before?’ I think the thing is, most of us rarely discuss how magnets work. And when we do, we use technical words (‘dipole’, ‘electromagnetic’, ‘electron shell’) which we and our listeners all know, which can obscure the fact that you don’t truly understand the underlying mechanism.

By all accounts this question, and the attempts to answer it, lead to an awful lot of magnetism-related discussions at scientific breakfast tables and coffee machines around the country. So, one outcome, of just this question, has been much thinking about and discussing the mysteries of physics, by scientists, with colleagues, and others, about a subject they all thought they understood, but actually it turns out they’ve got lots of questions about it.

If the essence of science is asking questions and taking nothing for granted, then I’d call this a result.

This post started life as part of a mammoth post I’ve been writing about how June’s I’m a Scientist event went. The post has taken about three days so far and we’re up to 3,000 words. So I thought I should really break it into bits and start bunging some up now.

Posted on July 13, 2010 modshane in Evaluation, IAS Event, Scientists | 2 Comments

A scientist gives us the lowdown on what it’s like behind the scenes…

As you all know, there are evictions every day for the rest of the week! You get a vote every day, so remember to keep voting for your favourite scientist.

But how do the scientists feel, being bombarded with questions and then facing eviction? One of the scientists who took part in I’m a Scientist in March, the lovely Louise Buckley, has agreed to tell us about what the scientists taking part are REALLY thinking…

    Louise Buckley

Hey there! I am Louise one of the scientists from the last “I’m a Scientist….” event. This competition is looking even more exciting than when I took part – I cannot believe how many questions are getting asked. It’s a total mash-up of questions – these scientists are going to need to take a holiday afterwards to recover!

I bet the scientists are starting to feel nervous now about who’s going to get evicted. We get labelled ‘the scientists’, but we are only human too. I know I was worried that you guys wouldn’t like me, that I’d get evicted because my science was crap or you thought I was a weirdo. Nobody likes to be evicted first. It’s kinda embarrassing. A bit like being a member of McKinley High’s Glee team 🙁

LMAO – my major fear was that you would ask me loads of questions that I didn’t know the answer too. I was terrified. And you did – and I coped and I learnt loads along the way. I can see from the questions asked at the moment that the scientists at the moment are getting asked loads of challenging questions. Keep ‘em coming. It reminds us that there is soooooo much more to science than our narrow fields of research. We sometimes forget that.

The scientists will be learning at the moment that it is okay to be wrong or not know the answers to every question. Sometimes you guys will know more than us, other times we will know more. You guys taught me that rainbows are circular – and I was like, nah, rubbish, wind-up, but it was true. Today’s scientists are learning all sorts of new things and they are learning them because of you.

But the most important thing that the scientists are learning at the moment is how to communicate our science to young people. Many of us don’t get that opportunity often. I know I was excited but scared that I wouldn’t be able to describe my work in a way that wasn’t total gobbledegook. These scientists are trying to manage it too.

We need to learn how to make our stuff sound interesting. So if you don’t think they have made something clear,  ask and ask and ask again. Don’t “Yeah, checked out of this conversation a minute ago…” (sorry Sue Sylvester!) Give ‘em feedback, show interest, and you can make today’s scientists better. Remember they are all terrified of being evicted so DON’T be too tough on them personally – but DO give them a hard time scientifically. And that’s how Sue Louise sees it!

Posted on June 23, 2010 modemily in Evaluation, Event News | Leave a comment

Busy, busy, busy!

Well the first week is almost over already and it’s been incredibly hectic. This is already the busiest event we’ve ever run – we’ve had nearly 4,000 visitors, just in the last few days!

Thank you to all the students for the interesting, funny and thought-provoking questions. And thanks to all the scientists for all the hard work they’ve put in answering them.

Students – feel free to comment wherever you want to on the site.

Visitors – although you can’t comment, you are very welcome to have a look round and read everything.

If you want to you can follow us on twitter – our twitter feed is http://twitter.com/imascientist, or the hashtag for the event is #IAS2010 – scientists and teachers, if you are on twitter, please use the hashtag if you are tweeting about the event, then people can find all the comments together.

We are having a whale of a time, we hope you are too. Best of luck everyone!

Posted on March 18, 2010 modemily in Evaluation, Project News | Leave a comment