Category Archives: Evaluation

Which browsers and devices are schools using?

Every year (see 2014) we take all our data, and look at how schools use the site; looking at how visitors access the site and how much that has changed in the past couple of years.

We’ve made a lot of changes in recent events, especially when it comes to registering accounts and using the site on mobile devices. We now pre-register all teachers and students, giving them usernames and passwords, so they can start asking questions immediately.

We can assume that the student visitors give a fair reflection of the general school IT facilities and system capabilities.

The graphs show the student data represented by dashed lines.

Device

We saw last year, that visits from mobile and tablet devices were increasing. and no surprise they continue to grow, while desktop usage drops slightly. For student tablet users there is little difference in content visited when compared to student desktop users.

We’re currently right in the middle of an overhaul to make the site more responsive and mobile friendly- which should make the user experience a lot smoother for all users. Allowing students who use tablets and mobiles the same experience as desktop users.

Browser

If desktop usage is going down somewhat, it’s unsurprising that Internet Explorer is also sinking,  with Chrome taking a the lead and Safari jumping up (a lot of tablets and mobile devices being Apple products, where the default browser is Safari). The rise in other browsers will be down to mobile and tablet browsers using Android, Blackberry and Opera.

IE versions

And finally the bane of web designers and developers lives.. The old versions of Internet Explorer. A collective sigh of relief, IE6 has finally drifted out of favour (available since 2003!!!!), IE7 is on the out, IE8 is dropping, as is IE9, even IE10 is dropping.. But IE11 has taken a big leap. Understandably when a new version is released, it will lead to older versions not being used, but the jump in the graph is still quite dramatic, compared to last years.

Posted on October 30, 2015 modemily in Evaluation, School | Comments Off on Which browsers and devices are schools using?

Research vs Evaluation

Science Learning+ is a significant funding scheme provided jointly between the Wellcome Trust and the National Science Foundation.

Learning can happen anywhere and at any time. Science Learning+ is an international initiative that aims to understand the power of informal learning experiences inside and outside of school.

The second aim of the scheme is to

“bridge the practice and research gap”

At a seminar in July aimed at providing an update on the Phase I project an interesting conversation developed about that gap between Science Communication practitioners and researchers.

I heard one speaker talk about practitioners wanting to know if a hypothetical red headline would give a 3% uplift in visitors. I responded on twitter:

Not all practioners agreed with me. Some felt each project would be unique enough to warrent a rewriting of expectations

Others simply disagreed and place efficacy as something for researchers:

In the end 140 characters felt underpowered.

For me research and evaluation are different, but very related.

I expect research to tell me if an approach to science communication works and how it works. I expect evaluation to tell me how well a project is working and how it can be improved. I would like evaluation to draw upon the research to extrapolate that particular  activities will lead to particular outcomes.

For example using I’m a Scientist:

The feedback we get from our participants is that connecting online with scientists improves their attitude to science, and to jobs in science. We seem to find the changes in attitude among girls is greater than it is for boys.

I want some research to tell me why those conversations are improving attitudes and if those changes are persistent. I want the research to be telling me how online activity compares to offline activity and why.

I want research to tell me what characteristics of engagement deliver the best and most persistent improvements in attitude and achievement.

Then I want my evaluation to examine our work against those characteristics and to suggest ways to improve them.

Research  = why something works

Evaluation = how well something works

What do you think?

Posted on August 27, 2015 ModShane in Evaluation, Science Engagement | Comments Off on Research vs Evaluation

Harwell Open Day Zone Report

HRWL LEGO PosterOn Saturday July 11th the Harwell Research Campus opened its doors to the public. 10,000 people were expected to come and see some of the most complicated and cutting edge science equipment in the world. Hundreds of volunteers working with that equipment were on hand to speak with the public and show them what happens.

We created a Harwell Zone to allow visitors to text in questions. It was promoted using posters as above.

The logic was that with 10,000 visitors some would not get to ask their questions, or may think of it later, or possibly be too shy to ask in person. It was a family day.

Working with the over-worked (understatement klaxon just sounded) outreach team at Diamond and Harwell we set the zone up and promoted it to exhibitors.

It wasn’t a great success. A mere 14 questions were sent.

However in the interest of continual improvement let’s look at what worked, what didn’t and most importantly what would be do differently if given the chance.

What worked?

The technology did. Visitors were able to text a question, get a response to manage their expectations and a notification when someone answered the question. We also had an enthusiastic response from exhibitors who signed up. However the enthusiasm was far far greater amongst I’m a Scientist alumni. It seems that we didn’t really reach many new scientists. It was in the main scientists we already knew who happened to be already exhibiting.

What didn’t work?

We didn’t reach new scientists. The vast majority of scientists there were not aware that they could have taken part. This meant that the online offering was very patchy. As you wandered around the trained eye (mine) occasionally picked out a poster. The only exception was the RAL Cyrogenics Lab where Vicky Bayliss had printed out extra posters and placed them all around the lab.

We didn’t reach the public. There simply was not enough publicity.

What would we do next time?

  • An online Q&A offering needs to be an integral and supplementary part of the open day. Exhibitors should be opting out not in. Taking some questions online should be seen as part of the overall experience. We need to brief potential participants better.
  • The online experience needs to match the offline experience. At Harwell some participants were running a desk, others signed up the entire lab. It was disjointed. It would work at the lab level better.
  • We should pay more attention to the offline visitor flow. At Harwell many labs had a distinct flow from start to finish. We should have been making sure as people left they not only handed back their lanyards but took away a leaflet offering the chance to ask follow-up questions.
  • Use the online zone to supplement the offline experience. At Harwell visitors were invited to view the Diamond Light Synchotron but there were explainers in there. It was a prime spot to publicise the zone.
  • Programme notes. People tend to keep the programme notes with them throughout the day and on the way home. That’s the time to mop up any unanswered questions.
  • Make more of the online zone. We could take feedback, promote new open days, communicate campaigns, point to more resources. The point is that if someone asks a question online they are in effect asking to engage in conversation. Same as offline. The difference being that online you have the ability to invite the visitor to restart that conversation at any time in the future.

 

Posted on August 24, 2015 ModShane in Evaluation, News | Comments Off on Harwell Open Day Zone Report

Engaging under-served audiences – talk at Bristol Zoo

Last week I was asked to speak about our experience of engaging under-served audiences. Here are my notes from which I spoke:

IAS is free online activity that connects school children and scientists. Kids go online, read scientist profiles, ask questions, take part in live chats and vote for the one they want to win.

Split into zones of 5 scientists and about 350 students. Mixture of general and themed zones. Been running since 2008 and reached nearly 70,000 students and over 1,000 scientists and engineers have taken part.

Scientists hear about it through colleagues, organisational emails and social media. They apply for the chance to take part. Tough selection process 1:4 for general zones.

Students hear about it through their teachers who sign up to take part. Over subscribed for I’m a Scientist and many teachers don’t get as many classes as they would like.

One way we do reach students who don’t normally get involved in science engagement is through product design.

Anecdotally, if a scientist visits a school, a class, 1:3 kids might stick their hand up to ask a question. With the online, pseudonymous nature of the activity we tend to get nearly 90% of the students actively participating – asking questions, chatting, commenting or voting. A side effect of this equality of voice is that not only do the quieter pupils get to ask their question but the other, louder, more confident students learn that their quieter peers do have something interesting to say. On a micro-scale this is reaching new audiences. And it is important. Not every engagement style works for everyone.

And it seems to suit some young people who don’t always get a say.

IAS is a spin off from a different project called I’m a Councillor, Get me out of here.

The inspiration for that project came out of conversations with council officers trying to interest young people in local democracy. They were doing things like inviting kids to shadow councillors for the day or they would invite students to visit the council chamber – and marvel at the majesty of our democratic overlords. The idea of an online activity was attractive to many, even if for some it just meant they weren’t going to need to clean up the graffiti left behind in the council chamber.

The why for us was that councils were trying to engage with mini-politicians, not mini-citizens. For me that was the more important democratic objective.

Over the 7 years that project ran some of the most memorable encounters were between the councillors and students from traditionally under-served groups.

I could talk about the pupil from the special school asking about the “futcha” for Bexhill or the feedback from the teacher at the PRU in Derbyshire who told us it was the first time that their pupils had ever been taken seriously by an authority figure. But it was the live chat between Nadia and a couple of Cardiff City Councillors that I remember best. The chat had been booked for 8pm by Steve a youth centre manager in Cardiff. It wasn’t the greatest of chats. Nadia was the only person online and was demanding a swimming pool for her and her mates. The two councillors patiently explained that Nadia should get together with friends and start a petition. After about 20 minutes of chat punctuated by strings of ****’s as Nadia’s choice of words got caught by our profanity filter, the chat was brought to a sudden end with Nadia typing, Nah can’t be bothered, bye bitches.

A disaster I thought.

5 minutes later Steve, the youth centre manager called me. That was brilliant he told me. Nadia had been excluded from every school in Cardiff and banned from every youth group. That night’s chat was the first time that Steve had ever seen Nadia engage with anything. Normally 1 in 5 words was a swear word but tonight in was only 1 in 20. He was ecstatic.

The point is that for some groups online communication is better than face-to-face and not just because of geography.

But having established that a project is good for certain groups it isn’t always as simple as that to get them involved.

We also look at the promotion aspect of reaching under-served audiences. Traditionally and anecdotally we’re told that science engagement projects do tend to reach the better resourced schools where kids often have plenty of privilege. Which means some schools are not taking part. But it isn’t so easy to identify them. Tied in is the issue for universities of widening participation as part of their fair access agreement required to charge £9,000 per year.

But what does widening participation mean? Apparently no two universities share the same definition (perhaps this has changed by now). We looked at a number of measures: IDACI, POLAR (participation in local area), and GCSE 5+ – when we looked at how the schools participating in IAS compared to the national profile against each of these criteria we were quite happy. We were getting a broad cross-section. And when compared to another large scale science engagement project we were performing well. But still not well enough.

There are flaws in the use of these measures. IDACI and POLAR give results based on the postcode of the school rather than it’s catchment area. The two can differ greatly. Secondly many of the schools in the poorest inner-city areas would also get greater funding through pupil premium and city-weighted education funding, perhaps allowing teachers more opportunities to bring in outside science engagement activities. Combine that with the fact that inner-city schools are also more likely to be within a short distance of a university and perhaps the inner-city schools aren’t so underserved.

So we still have a challenge on defining underserved.

We’re currently trying to work with other organisations like STEMNet to form a definition but in the meantime one of the factors we are looking at is distance. We’re looking to target schools from the poorer coastal areas in the UK.

But targeting is easier said than done.

We’ve conducted a few small experiments in recent years.

1. For one zone recently we sent 20 secondary schools in the most income deprived areas of the UK a teacher pack for the event inviting them to participate in the upcoming event. The idea was to make it as simple as possible for them to join in. Not one of them did. Not great.

2. More recently we posted a letter and some flyers to 200 primary schools in rural and coastal areas inviting them to apply to take part. The letter explained that Y5/6 was particularly important and that since they were in a remote area they might gain the most benefit. We’ve had 2 sign up. Not bad. OK

3. For a Food Zone in 2013 we looked at the schools signed up for the zone and worked out which would “qualify” as widening participation schools. There were about 5. Each teacher was called to make sure that they had received their packs and that they understood how the event worked. We looked after them, gave them special treatment. Every single one showed up online compared to a usual 2/3rds rate. It worked.

The learning we take from this is that you should look after the people already showing an interest. You get a better impact than simply going after more and more of your target. It might sound obvious, but for us we don’t have the time to call every school.
But we now realise that the time and money we might have spent sending stuff out to schools who hadn’t shown an interest, would be better spent on looking after the ones who had already shown some interest.

We’ll be repeating the specific targeting of schools in remote areas and giving special attention to those who respond. Thank you.

Posted on June 8, 2015 in Evaluation, Science Engagement, Widening Participation | Comments Off on Engaging under-served audiences – talk at Bristol Zoo

March 2012 – March 2015 Evaluation Report

We were awarded a Wellcome Trust Society Award to run I’m a Scientist from March 2012 until June 2014. We later received a grant extension for November 2014 to June 2015. This report is all about our learnings in these last three years: from March 2012 until March 2015.

Our main learning points are:

WT report cover

Click here to download the full report

1. I’m a Scientist has gone from 30 zones per year in 2012, to 54 zones scheduled in the 2014/2015 school year.
2. Expanding zones to different audiences: primary school students and general public shows.
3. I’m a Scientist is a public engagement boost for scientists.
4. I’m a Scientist gets students enthused about science.
5. I’m a Scientist reaches a diverse set of students.
6. Teachers come back, but tricky to track.
7. Students ASK about cancer, animals, and life and CHAT about science, scientists and work.
8. Moving forwards, further adaptation to new technologies (such as tablets and smartphones) is important.

Click here to download the full report.

Posted on May 26, 2015 modangela in Evaluation | Comments Off on March 2012 – March 2015 Evaluation Report

Understanding Animal Research – Google Hangout

UARBefore our most recent June 2014 I’m a Scientist event we partnered with Understanding Animal Research to run a Google Hangout for scientists who work with animals.

Students often ask the scientists for their views on animal testing, and we want to help give scientists the confidence to be open and talk about working with animals in their research.

John Meredith, Education Manager at UAR, ran this session with six scientists who were taking part in I’m a Scientist the following week:

The A word: how to talk about animals in medical research 
This informal webinar presented by Understanding Animal Research will look at how and why we should talk about animals in medical research. It will cover current public attitudes to animal research, the facts and figures, the value of openness and how to answer the tricky questions or deal with confrontation. There will be plenty of time for questions and discussion during and after the presentation, which should last around an hour.

We asked the scientists to fill in a short survey after they’d taken part in I’m a Scientist, to assess whether the Hangout was useful, and whether we should offer it before every event.

Five scientists gave very positive feedback. In summary they found the Hangout useful and informative, but often didn’t use the training in I’m a Scientist as students didn’t ask about animal research, and the scientists didn’t want to bring it up. If we run it again we could make it more interactive, giving the scientists more practice in answering potentially difficult questions during the session.

Was the Google Hangout good?

  • All 5 scientists said yes

Did you use the training during the event?

  • 3 said no, because it didn’t come up, and they didn’t want to mention it unless they had to
  • 1 said yes, finding themselves using the training many times during the event
  • 1 said a little, but they didn’t get many questions on it

Did it make you more confident to talk about animal research?

  • All 5 scientists said yes
  • 1 of the scientists mentioned they were wary of talking about animal research in the live chats, in case time ran out and they couldn’t explain their work properly

Would you recommend it to others?

  • All 5 scientists said yes

How would you improve it?

  • Open the session up and ask for individuals input more
  • Links to resources that show that animal experimentation is not ‘animal cruelty’
  • Have more scientists who use animals in their work, to encourage more open discussion among peers

And a few other comments:

“Very helpful and answered a lot of the questions I had about discussing animal research with the public”

“I have contacted UAR and organised a school visit of my own!”

Posted on August 6, 2014 in Evaluation, IAS Event, News | Comments Off on Understanding Animal Research – Google Hangout

Wellcome Trust engaging science day

How to share and disseminate the learning from your project and evaluation.

On 14th May the Wellcome Trust held one of their twice-yearly Engaging Science days. They’re an opportunity for grantholders to meet with each other and Wellcome staff, to share learning and ideas. And to sample Wellcome’s renowned lunch offerings (side salad complete with edible flowers and quail eggs, anyone?).

Gallomanor were invited to speak on the panel of the session convened by Ben Johnson of Graphic Science, with the blurb “Once you have completed your project how do you widen its impact by ensuring others learn from what did (and didn’t) go right? An effective evaluation can also help with securing further funding and a wider roll-out of your project.” Other panelists were Manisha Lalloo from the Royal Academy of Engineering giving a funder’s perspective, and Becky Parker & Dave Colhurst from Simon Langton Grammar School.

Evaluating the learning from our projects and disseminating it (both within the office to colleagues and to the wider SciComm community) is important to us, so this is a summary of what I spoke about in the 5-10 min slot:

Capture

Angela sums up my talk excellently

  1. Be brave in budgeting. If you think an external evaluator will help, budget for it. Plan honestly for the amount of staff time your evaluation process will take. 
  2. Write up and talk about your findings, however small they seem. We all have to report back to funders, but making these reports readable will encourage others to read them. Start with a concise executive summary, use visual cues to break up text. Beyond this, share what you’ve learnt with colleagues and peers. We have an evaluation site where we post snippets of learning. Think about who you’re writing these for and target them – funders, participants (scientists, teachers, general public), or practitioners?
  3. Plan viable dissemination routes. Be innovative with evaluation and people will notice and ask you to speak and share learning at conferences. We’ve found this a brilliant way of publicising our projects, and it’s bought in new funding from people in the audience, such as the online discussion zone for the Ri Life Fantastic CHRISTMAS LECTURES.

And my top tip for dissemination when Ben sprung the question on us? Be short and visual in reports.

Posted on May 19, 2014 in Evaluation, News | Comments Off on Wellcome Trust engaging science day

I’m a Scientist zones for primary students

We’ve had plenty of primary schools take part in I’m a Scientist zones in the past, nestled in among secondary schools. Feedback from primary teachers and students shows they get a lot out of taking part. Primary schools are looking for science enrichment activities too, and talking with real scientists is exciting at primary school as well as secondary. Scientists often wouldn’t know the students were Year 5 & 6, not Year 7 & 8, if we didn’t tell them.

caesium

This March we decided to run 2 primary school only zones, for Year 5 & 6 students. In part to avoid possible situations of primary students reading questions on non-primary-friendly topics asked by secondary students in their zone (think sex, drugs, rock’n’roll). Partly to open up I’m a Scientist to a wider group of UK schools and students. And importantly because one of the conclusions from the ASPIRES project about young people’s science and career aspirations, is that STEM education projects need to begin earlier, at primary school.

xenon

The 2 primary zones – Caesium Zone & Xenon Zone – were general zones, each with 5 scientists from a range of research areas, but avoiding any non-primary-appropriate topics (no IVF experts for instance).

What we learnt:

  1. There is demand for primary zones. We had too many teachers wanting places and had to turn some away. We’ll be running more primary zones in future events. 
  2. Primary zones aren’t that much different to secondary zones. They felt like just any other zone. We don’t need to change the format for primary students. The main difference we saw was that the questions were often more factual than conceptual, live chats were less disruptive than with older students, and students left lots of comments thanking scientists for their answers.
  3. Being in a primary zone didn’t seem to affect the scientists’ experience of taking part. When we offered them a place we mentioned it was in a primary zone but that shouldn’t change how they approach engaging with the students.
  4. Online safety is more of a worry for primary teachers and parents. We need to be clearer about how secure the site is and giving advice, such as students not using their first and last names in their username. This is true for secondary zones, too.
"Did you always like science when you where in primary school"

“Did you always like science when you were in primary school”

Posted on April 10, 2014 in Evaluation, News | Comments Off on I’m a Scientist zones for primary students

How does I’m a Scientist affect students’ attitudes to science?

We know anecdotally that participating in the I’m a Scientist event has a positive effect on students’ attitudes to science. Teachers tell us that their students were buzzing, and that they understood more about science. Students thank scientists in live chats for an interesting lesson. They tell us that it is “better than Facebook”.

But being data geeks that isn’t good enough. We wanted to know just how much we were affecting the students. Were we affecting them all by the same amount? Girls, boys, year groups? Did being more active in the event mean a greater change in attitude?

Questions we ask students

Thanks in part to the pilot work by Robin Longdin, then a SciComm masters student at UWE, we ask every student registering for I’m a Scientist a set of 4 questions:

  • How does school make you feel about science?Registration Questions
  • Are you planning to choose a science subject at the next stage of your education?
  • Do you think jobs involving science are interesting?
  • When you finish your education, how likely are you to look for a job that uses your science knowledge and skills?

We also ask the same questions towards the end of the event and ask for the username so we can match against their initial answers, their gender, year group, zone and activity levels within the event.

Measuring attitudinal change

The important thing for us is not the answers they give but the difference between the beginning and end of the event. We want to know if after two weeks of being exposed to our scientists they feel differently.

Each question had a five point answer scale. We intentionally wanted a neutral middle answer. We can’t expect students to all feel passionately about science. We then allocated a numerical value to each answer. 2 for the most positive, -2 for the most negative. We then simply subtract the starting answer value from the end of event value to give us a value for change in attitude for each of the four questions for each of the student who answered.

Do more active students see a greater change in attitude?

There is a lot of data generated and one of the challenges is to consolidate it into meaningful clusters. For example what do we mean by levels of activity? Some students are lively in live chats. Others ask a lot of questions. Some do both. We looked at the 3 main types of activity: ASK, CHAT, VOTE and scored each student between 0 and 3 depending on how active they were. We then totalled the individual scores to give each student a score between 0 and 9 overall. That is the X-axis below.

The first thing to note is that the overall trend is that the more activity on the site the more positive the change in attitude. Secondly the change in attitude to the final question: “When you finish your education, how likely are you to look for a job that uses your science knowledge and skills?” is much lower than the other questions, and in fact sometimes negative even for those students who made the most of our erstwhile role models.

We need to investigate this further. It could be a flawed question? What does it really mean to 13 and 14 year olds. Do school students consider a PhD studentship to be a job? Are half our role models not helping with that question?

UPDATE:
We went back to the source data and found to our embarrassment that we had made a mistake and we were comparing the answers to the final question to the answers to the 3rd question. D’Oh! The good news for us is that when you use the right data, the story looks even better. Students emerge from I’m a Scientist feeling much more positive about jobs in science for themselves.

The greatest effect is on girls and Year 10 & 11 students

This graph shows the average change in attitude for 6 groups by question asked. It clearly shows that the event has the greatest effect on girls and on students in Years 10 & 11.

This is based on a total of 853 valid responses from students across 3 events from June 2013 to March 2014 in the UK. The n for the groups in the second chart are:
All: n=853
Girls: n=475
Boys: n=333
Primary: n=90
Years 7-9: n=589
Year 10 & 11: n=130

They don’t all add up to 853. Sometimes it is a group of girls and boys who register. We also had 21 6th formers respond, but that is too small a sample to use. And sometimes the default “Please choose” answer remained. Yes there are disproprtionately more girls than boys, but yes, more girls than boys do participate.

Leave a comment and start a conversation!

Finally, we do this analysis for our own sake so that we can better understand our event, but we publish it for your sake, whoever you are. We want other practitioners to share in our knowledge. Please leave a comment to say “thanks”, or to ask for clarification or for more information.

Posted on April 9, 2014 ModShane in Evaluation, News | Comments Off on How does I’m a Scientist affect students’ attitudes to science?

March, June or November?

When teachers ask me what aged students they should bring on I’m a Scientist, my answer is often something along the lines of “well, it’s developed for Year 9 students but all ages get something different out of taking part, so take your pick”.

The good news is that our data supports this. When students register we ask them what year group they’re in. There are Year 5s. There are also Year 13s. Around two thirds of the 30,000+ students registered so far are KS3 (Years 7, 8 & 9).

The bad news? It’s not really bad news as such, but it looks like I should also be advising teachers on the time of year they should take part with different year groups.

March and June are great for pre-GCSE students. Teachers tell us they’re looking for something fun to do with students at the end of the year. They’re less good for GCSE and A Level students who have mocks, coursework and shock horror, actual exams to contend with.

November is slap bang in the middle of careers time for older students, and who better to speak to about possible careers in science than 5 practicing scientists online?

Different year groups take part at different times in the year

Different year groups take part at different times in the year

And just in case you’re wondering why there are a suspiciously high % of Year 5’s – that’s the registration form’s default option. So you might even expect it to be higher!

Posted on March 14, 2014 in Evaluation, News | Comments Off on March, June or November?

Girls vs Boys

I was asked recently what the gender split was for students taking part in I’m a Scientist. I replied that we’d never looked on the assumption that since teachers took their students online it would reflect the school population.

I’m never one to turn down the chance to crunch some numbers so I took a look. And was surprised. At registration we ask if the person registering is a Boy, Girl or A group of students (where more than one pupil is sharing a log-in). We have data going back to June 2012 and nearly 15,000 respondents to the question.

IAS Gender Split

Surprisingly it wasn’t a straight 50:50 split. So I then decided to look at it by year group.

IAS Gender by Year Group

What we saw was that year 9 & 10 were significantly skewed towards girls. And that bias continued through to 6th form.

Is that common for science outreach events?

Posted on March 11, 2014 ModShane in Evaluation | Comments Off on Girls vs Boys

I’m a Scientist so far in numbers

How many zones have we run? How many students engaging with how many scientists? Asking how many questions? What year group are they? Where in the UK are they?

The first slide of the I'm a Scientist summary so far...

The first slide of the I’m a Scientist summary so far…

These are some of the questions we get asked a lot, and until now haven’t had all the answers in one place to roll off the tip of our tongues. We’re pulling together some slides summarising I’m a Scientist to date and once they’re done we’ll post them up here.

In the mean time, I’ll be posting snippets that show just how far we’ve come since our first big Wellcome Trust grant in 2010.

To start us off (and to set the scene for the graphs and maps to come in future posts) here are some of the numbers so far, up to February 2014:

Zones: 118

  • 51 general science zones, 67 themed zones
  • Spread across 10 events, in March, June & November each year

Scientists: 590

Students: 36,225

  • % of students that actively took part: 84%
  • Average number of students per zone: 307
  • From 462 schools

Number of questions asked: 86,645

  • Number of questions approved: 38,836
  • Number of answers given: 84,890

Live chats: 1,616

  • Lines of live chat: 528,035

Visits to imascientist.org.uk: 1,592,016

  • Unique visitors: 1,306,978
  • Page views: 4,958,725
  • Average visit duration: 3:02 minutes
Posted on February 14, 2014 in Evaluation, News | Comments Off on I’m a Scientist so far in numbers

Visits to IAS by technology type

As part of a review of what we need to do over the next 5 years with I’m a Scientist, we looked at how visitors accessed the site and how that has changed year by year over the last 4 years. We also looked at how visitors registered as students differed from All visitors. The Student visitors probably give a fair reflection of school IT capabilities. In the graphs the student data is represented by dashed lines. We only have student data from 2011 when we started tracking registered users separately.

IAS by device
It was no surprise to see that visits from mobile and tablet devices were increasing but having 23.6% of visits from mobile devices in 2013 was a shock. Delving deeper it is clear that it is mostly questions that mobile users visit. Tablets tell a different story. For student tablet users there is little difference in content visited when compared to student desktop users. And it is interesting to see that the levels of tablet use are the same for students as for all users at just under 8%.

We need to adapt our design for the question page so mobile users get a better experience.
We also need to be more aware of how easy it is for registered tablet user to interact with the site.

IAS by browser
Browser usage tells an interesting story. Unsurprisingly Internet Explorer is plunging. But at over 60% it is still the most popular browser in schools. More surprising was the drop in usage of Firefox. The rise in other browser is mostly due to mobile and tablet browsers with Android, Blackberry and Opera featuring.
IAS by IE
Finally the web developers enemy – old versions of IE. IE6 has just about disappeared. Yay. IE7 hangs on at 4%, but IE8 remains popular. The surprise for us in this graph was that All visits and Student Visits differed little in respect of IE version.

If you have any questions or would like more data. Please let us know in the comments or via email.

Posted on January 22, 2014 ModShane in Evaluation | Comments Off on Visits to IAS by technology type

I’m a Scientist acts as a public engagement booster for scientists

Anecdotally, we’ve heard of how I’m a Scientist can be a good starting point for science communication activities. This was the case of Suzi Gage, Tom Crick or Suze Kundu, who took part in I’m a Scientist in June 2011 and are now putting a great emphasis on the communication side of their scientific careers or even fully devoting to it.

Twitter conversation about the impact of I'm a Scientist on Science Communication careers

Twitter conversation about the impact of I’m a Scientist on Science Communication careers

However, we were still curious to know to what extent we could extrapolate this to the wider community of scientists that have participated in the event during the last years. How could we know if I’m a Scientist had encouraged them to do more science outreach? Well, we decided to ask them.

We sent a survey to all the scientists that had participated in I’m a Scientist until 2012, leaving a gap of at least one year since they took part in the event. Approximately a quarter of the scientist responded to the survey, resulting in a more than decent sample of 113 scientists. The data collected in the survey show that there is a strong indication that I’m a Scientist is a real public engagement (PE) boost for scientists.

Summary of the survey's main findings

Summary of the survey’s main findings

I’m a Scientist is a good way to start doing public engagement in schools

If we look at public engagement at schools, we have recorded a significant increase, especially among those scientists that were particularly new to this form of outreach. Moreover, scientists who had never done school public engagement were encouraged to do more public engagement in general, going from none to an average of over 4 activities per year.  This was supported by some great comments gathered in the survey:

Having never done outreach with schools before, IAS gave me the chance to engage with a different audience than I would typically.” – Scientist

Scientists find the event flexible, open and inclusive

The online nature of the event was praised by different scientists that left comments in the survey:

I really liked how the online format broke down barriers and allowed the students to ask anything they wanted without having to stand up in a crowd.” – Scientist

Having all the activity online also gave me the flexibility to contribute more of my time, ad from a remote setting, compared to face-to-face school visits.” – Scientist

"Sifting the evidence" Suzy Gage blog hosted by The Guardian

“Sifting the evidence” Suzy Gage blog hosted by The Guardian

Some of the scientists had already contributed to the online scientific community with blog posts, podcasts or through social media. However, for a high proportion of them (68 out of the total 113) I’m a Scientist represented their first online public engagement event, which really pushed up their participation in general public engagement. In this case, their collaboration with public engagement activities went from zero to more than 3 general outreach events per year.

It was also interesting to find out that those who had already done lots of public engagement (4-15 activities per year) started doing more specific online outreach after participating in I’m a Scientist, at the expense of other forms of public engagement.

I’m a Scientist reveals itself as a great launchpad for budding science communicators

The majority of the scientists that filled in the survey (86%) had already taken part in some kind of public engagement activity (lectures, science festivals, interviews in traditional media, science policy making, etc.). It was a nice surprise to find out that scientists who had done very little public engagement (1-3 activities per year) increased their activity dramatically, going up to 5-8 outreach activities per year after the event. What is more, the greatest increase in overall public engagement activity (an increase of 130%) corresponded to the scientists who were just doing very little of it before the event.

Looking at the big picture, there is a general trend that I’m a Scientist enhances the participation of scientists in outreach events, except in the case of those who already did loads (more than 30 activities per year), where there is very little room for improvement. All in all, we are happy to confirm that I’m a Scientist represents a great launching platform for public engagement in science.

It remains the best public engagement event in which I have been involved.” – Scientist

 

Posted on January 8, 2014 modangela in Evaluation, Event News, News, Science Engagement, Scientist Benefits, Scientists | Comments Off on I’m a Scientist acts as a public engagement booster for scientists

How does I’m a Scientist change students’ perceptions of science?

We’ve just run I’m a Scientist in Ireland and are curious to know how our event actually affects students’ attitudes towards science.

In order to do this, we included a short and compulsory pre-event survey in the form students used to register. We then asked students to fill in the exact same survey on their profile page after the event. When we matched the data from the two surveys, 92 students (7% of 1,247 students that participated in I’m a Scientist) had filled in both surveys. Importantly, data from the total number of students that filled in the pre-event survey very closely correlates with the pre-event data of this 92 student sample.

We were very happy to find out that students’ interest in science and science related careers is clearly increased after taking part in I’m a Scientist. This is what we have learnt:

The amount of students that say they love science doubled after taking part in I’m a Scientist

How does school make you feel about science?

How does school make you feel about science?

I’m a Scientist really got the students excited about science! Before taking part in the event, only 23% of students said they loved science, but this number increased up to 51% after the event. On the other hand, the number of students that don’t feel really excited about science or think it is boring decreased from 9% to 3%.

Participating in I’m a Scientist encourages students to choose a science subject in the next stage of their education

In the pre-event survey, 66% of the students were absolutely certain or very inclined to choose a science subject next year. However, this percentage raised up to 71% after participating in I’m a Scientist.

Thanks guys for talking to me really helped me make my decision for the leaving cert – sarahlawless, student

Students are keener on science related jobs after participating in I’m a Scientist

The majority of students that completed the surveys already thought that jobs involving science are at least fairly interesting before taking part in the event, but there was still room for improvement and the percentage of students that considered science related jobs very interesting saw a big increase from 36% to 62%.

Taking part in I’m a Scientist increases the likelihood of students looking for a job that uses their science skills

How likely are you to look for a job that uses your science knowledge?

How likely are you to look for a job that uses your science knowledge?

Before taking part in I’m a Scientist, a big portion of the students (37%) couldn’t decide whether they would try to look for a job that uses their science skills and only 10% said they were sure that they would look for this type of job. However, the event seemed to be the boost that students needed to be more confident about looking for a science related job. After I’m a Scientist, most of the students (68%) said that they would certainly or very probably look for a job that uses their science knowledge.

Honoured to have taken part. The future of science is in very good hands with you guys! – scientist

In addition to this, students left several comments that stated clearly how they were enjoying and learning at the same time throughout the event. They liked that the event was so interactive and that they had an active part at every step: asking, commenting, chatting and voting.

In the future, we would like to use this same strategy to measure the impact of other events or activities. We are also very interested in analysing the gender and year course differences that could be found in the pre-event data.

Posted on December 19, 2013 modangela in Evaluation, Event News, IAS Event, News, Science Education, Science Engagement, Teachers | Comments Off on How does I’m a Scientist change students’ perceptions of science?

IAS Live: Antibiotic Awareness Day

Yesterday afternoon, on European Antibiotic Awareness Day, we joined the Royal Society of Chemistry (RSC) at Burlington House in London, Piccadilly for a foray into antimicrobial research.

CAPTION

I’m a Scientist: Live at Burlington House.

We took five scientists — three of whom were I’m a Scientist alumni, having taken part in previous online events — whose research looks at the uses of antibiotics and antimicrobials, put them in a room in-front of around 100 sixth-formers, and had them answer questions on everything from Typhoid Mary, to zombie apocalypses.

Our scientists were Rob Shorten, Jess Bean, Mark Roberts, Clare Taylor, and Emma Newton.

CAPTION

Jess Bean and Mark Roberts go for the BAFTA puppeteering award with plush microbes.

Helen Arney, our MC for the afternoon kicked off by introducing the scientists before launching into a round of Science Fact or Fiction. The facts covered everything from bacteria flavouring cheese and yogurt, to our being only 10% human, with most of the cells in the body (by number) actually being bacteria. Clare introduced the notion that around 3% of the population can carry Salmonella their whole lives without ever becoming ill; setting up a whole round of questions on historical poisoning and Typhoid Mary.

We then moved on to the questions round: Emma presented a world first, showing us her “electronic nose”, a prototype device for sniffing out different bacteria based on the chemicals they produce. Using plush microbes, Jess and Mark acted out an Academy Award worthy demonstration of a bacteriophage’s attack on an E coli cell. Rob took us through the purposes of a faecal transplant, Mark explained how some bacteria can follow the earth’s magnetic field, while Clare took the audience award to the best piece of advice for surviving a microbial zombie outbreak: “there’s always a percentage of the population with a resistance; seek them out and make a cure.”

In the end there could only be one winner. After giving the scientists a last plea for votes the votes were counted yielding a surprisingly close spread, with Clare standing above the others to take hold the coveted I’m a Scientist: Live trophy and mug.

Let’s evaluate

CAPTION

Standard end of event competitors’ photo, accompanied by MC, Helen Arney.

The room was filled, with around 100 students in their seats, votes being counted using the electronic voting system were all around the 90 mark. The audience was made up predominantly of school students, with about 12% being teachers.

The use of electronic voting pads gave us the chance to do some on-the-spot evaluation of how the audience found the event. A majority of the audience who provided this feedback said they had learnt something with 49% saying they had “learnt a lot”, and 37% learning “a bit”; only 8 respondents (14%) said they had not learnt anything.

To the question, “Has the show made you want to find out more about antimicrobials?”, 50% of those who replied “got the bug”, while 18% had had enough. Owing to a slight technical fault we weren’t able to count the replies to, “Did you enjoy the show?”, what we did get were a whole lot of positive comments from students and teachers.

I'm a scientist live - RSC - Did you learn anything?I'm a scientist live - RSC - Has the event made you want to learn more about antimicrobials?
The audience seemed to really engage with the scientists, asking a good mix of in-depth scientific questions, as well as some more sillier ones. This was helped in no small part by Helen’s fabulous job setting the tone, demonstrating that there could be no such thing as a bad question; and putting the audience and the scientists at-ease.

Really informative; loved the bit when the scientists talked about their specialties; really interactive – the pads were great; great venue and they liked the refreshments (felt very adult!)

Many students stayed behind to ask more questions to the scientists once the event was over, and working by the buzz in the room, and the comments from teachers and students, it seems like the afternoon was a resounding success.

What have we learnt?

IAS-Live

  • Scientists bringing props is a good thing.
  • Being nice and altruistic can get you lots of votes, or certainly lots of applause.
  • It’s useful to run through the technical aspects of the event maybe a couple more times than might feel necessary; can’t be too prepared.
  • Students appreciate when a refreshment buffet includes cake.
Posted on November 20, 2013 Moderator - Josh in Evaluation, I'm a Scientist - Live | Comments Off on IAS Live: Antibiotic Awareness Day

IAS Live: Drugs, Bugs and Infections

Last night five scientists researching different aspects of antimicrobials took to the floor at the FIS Conference at the ICC in Birmingham to take part in I’m a Scientist Live: Drugs, Bugs and Infections.

Photo courtesy of @biologyatkinge1

Photo courtesy of @biologyatkinge1

Our five were:

Mark Webber

Richard Bax

Amy Wedley

Anne-Marie Krachler

Peter Hawkey

AMK & LP

Photo courtesy of @PharmaMix

Our MC for the evening, Simon Watt, kicked off by spending 10 minutes interviewing Laura Piddock, Chair of Public Engagement at BSAC about the current situation in research. The scientists introduced themselves and gave their Science Facts and Fictions before the 100-strong audience launched with their questions.

30 mins later with time running out, Simon brought the event to a climax by asking the scientists to spend 30seconds telling the audience why they should get the votes and the winner’s trophy. The vote that followed confirmed that Anne Marie’s promotion of the IMI’s Summer School for Year 12 students was enough to edge her into the lead and win the vote.


Some numbers:

IASLIVE FIS Who are you

181 tickets ordered.

105 attendees on the evening (105 voting handsets were used).

Over half the audience were students evenly split between schools and undergraduates. This means a lot of the students who asked for tickets failed to show. We had a capacity of 150 for the evening with a maximum limit of 200 for the room. Whilst we expected some dropout we are surprised by the level of no-shows. We will need to over-book more heavily in future.

Voting was close. Only one vote separated Anne-Marie from the scientist who was second-placed. We discussed what to do in case of a tie beforehand, but were clear it wouldn’t end up as a draw. We need to take those plans more seriously in future.

Having the electronic voting cards allows us to ask some immediate evaluation questions. A healthy majority enjoyed the show and learnt from our panellists. A slightly smaller majority have caught the antimicrobial bug.

IASLIVE FIS MoreIASLIVE FIS Learn


What else did we learn/have confirmed from last night?

IASLIVE FIS Enjoy

  • Allowing a 30 minute turnaround between a lecture and IASLive with only one sound engineer is a little optimistic.
  • We should brief participants that they will be using a lapel mic and that they therefore need a lapel.
  • Some sort of stage helps if the audience are on the same level
  • Enthusiasm and energy will give rise to a decent discussion whatever the subject.
Posted on November 20, 2013 ModShane in Evaluation, I'm a Scientist - Live | Comments Off on IAS Live: Drugs, Bugs and Infections

What criteria do students judge scientists on?

When students take part in I’m a Scientist they get to vote for their favourite scientist to win £500 to spend on communicating more science. It gives them ownership of the project and they decide who gets, in effect, a small grant for public engagement. But what are students’ votes based on?

This is how 855 students (or groups of students) have ranked certain criteria from most to least important when considering how to vote for scientists, since January 2012. The results come from a Drag & Drop ranking activity in the first lesson plan “You’re the Judges” that teachers run to introduce their students to I’m a Scientist.

Voting ranking criteria

The top ranked criteria are all linked to health or the scientific method. This is reflected in the number of questions students ask scientists about saving lives and animal testing. Reassuringly, the more superficial criteria lurk at the bottom of the list.

Other criteria that students suggested include:

“Abides to Scientists ethics, and morals”

“A person that thinks ‘outside the box’. Confident person. Optimistic person”

“Enthusiasm”

“How much they believe in and care about their work”

“The speed that they answer my questions. Also if they turn up for a chat”

“How great their beards are”

“If the scientist works at weekends (as well as during the week)”

And finally, the crux of all scientific research:

“Whether their tests are fair or not”

One teacher commented that, “the class disagreed on the ‘good-looking’ criteria because some feel that people are judged by first impressions and looks are included in that”.

Teachers, how do these criteria compare to how your students judge the scientists? Did students generally agree on the rankings or did certain criteria provoke more discussions than others?

Posted on November 8, 2013 in Evaluation, IAS Event, News, Teachers | Comments Off on What criteria do students judge scientists on?

Using teacher feedback to plan future themed zones

A question I sometimes get asked when I tell people about I’m a Scientist, is how do we choose which zones to run? In short: we ask teachers what they’d like.

In July we emailed all the teachers signed up for I’m a Scientist to ask them what zones they’d like to see in our next three events: November 2013, March 2014 and June 2014.

Just over 40 teachers filled in each survey by ticking as many of the 40 zone choices as they’d like (some are zones we’ve run before, others are new).

Which zones came out top?

20 zones

The graph shows the top 20 zones, by the number of teacher requests across the 3 surveys for November, March and June. Forensic Science and Genes zones came out top, closely followed by Stem Cells, Health, Organs and Sport Science. Most of the top requested zones are biomedical; this might be because there are more Biology teachers on our list, due to the majority of funding coming from the Wellcome Trust since 2008.

When to run these zones?

What’s more interesting is the variation when teachers want to see certain zones, reflecting when topics are taught in the school year.

  1. Stem Cells, Drug Development, IVF and Subatomic zones were more requested in November than March or June.
  2. Earth, Chemicals and Technology zones were less requested in March than November and June.
  3. Communication and Human Limits zones were more popular in June than November or March, while Blood and Reproduction zones were less popular.

So what…?

We’ve used this feedback to choose some of the zones we’ll run over the next year in advance. This will help us plan teacher and scientist recruitment throughout the year. STEM contract holders have also told us it’s useful to know dates and zone themes in advance.

And sad news for any linguists out there, I don’t think we’ll be running a Language Zone any time soon. With only four requests there just isn’t the demand.

A taste of the themed zones running over the next year is at imascientist.org.uk/2013/10/im-a-scientist-zones-november-2013-june-2014. We’ll add more themed zones from other funders to the mix nearer to the events.

Posted on October 30, 2013 in Evaluation, IAS Event, News | Comments Off on Using teacher feedback to plan future themed zones

We’ve published all our zone reports from June

We’ve published all our individual zone reports from the June 2013 event. If you delve further back into the archives, you’ll find them for the zones we ran in March 2013 & November 2012 too. All the reports are tagged (funnily enough) under “Evaluation Reports“.

Each report sums up the activity in that zone. They provide some information for the scientists and teachers in that zone about how much the students and scientists interacted.

How many questions were asked and answered? What were the popular topics asked by students? What examples of great engagement stood out? What did students, scientists & teachers say about taking part? How does the winner plan to spend their £500 on communicating more science?

We email out each report to all the scientists and teachers in that zone. Scientists tell us the reports really help justify their participation to supervisors and colleagues. The detail tells them far more than just receiving a participation certificate in the post. The ‘loads’ of questions they answered turns out to be in the hundreds. The live chats went in a flash thanks to around 300 students bombarding them with questions over the 2 weeks. And the students really did ask about their research – volcanoes, bacteria & cancer for the zone below!

Technetium wordle

Take a look. If you’ve got any comments, such as what other info might be useful for scientists and teachers, we’d love to hear them.

Posted on July 17, 2013 in Evaluation, News | Comments Off on We’ve published all our zone reports from June