Category Archives: Evaluation

Gender differences in online engagement

“I thought scientists just looked like they do in the film Flubber and experimented on aliens or weird stuff but when I found out you liked Taylor Swift I realised you are more down to earth and not like mad scientist :)” – Student, November 2015

I’m a Scientist is about connecting pupils with real scientists. Something we want to find out is what effect this interaction has on different groups of students. Recent research by the Institute of Physics has done just that.

The IOP has carried out an independent research project to gain an insight into the behaviour and attitudes of boys and girls who do I’m a Scientist. As part of their Improving Gender Balance project, the IOP funded two zones: Terbium Zone and Osmium Zone.

The researchers anonymously surveyed the students’ attitudes to science and scientists before and after the event. We also provided them with all the student’s interaction data from the sites. Every line of live chat, every question in ASK. Specifically the researchers wanted to look for gender differences in two areas:

  1. How taking part in the event changed students’ attitudes to science and scientists, and
  2. The types of questions that students asked the scientists.

The first results from analysis of this treasure trove are now online at the IOP blog.

IOP gender difference Q1

Physicist, know thyself: After the event, girls in particular reduce the negative words they use to describe you.

Check out the post for their findings on the effects of taking part in I’m a Scientist, including:

  • positive changes in word choice to describe scientists
  • an increase in girls’ awareness of careers where Physics A-Level is useful
  • girls’ improved confidence in talking to physicists

Girls want to see the scientists as real people

The report also dissects what girls and boys want to know from the scientists. Interestingly, as well as asking more questions overall, girls seem to ask many more personal questions, asking more than boys about job satisfaction, the career goals and achievements of the scientists, and their motivation to work in science.

For us, the identified trends make sense. For example, a boisterous classroom culture can be a barrier for quieter children to engage with visitors. As Natasha at the IOP notes, ‘the online, anonymous nature of the live chats gives students more freedom to ask questions than a traditional careers talk or even a speed-networking-style careers event.’ Girls, as a group, seem to relish this freedom.

And we believe that for science to be appealing there’s no need to glam it up with explosions and ‘wonder’. The more students hear about the reality of science from people like them, the more positive they feel about it. This research backs up that view.

Research with us

If girls are a group that benefits from this increased exposure to scientists as real people, it’s likely other underrepresented groups do too.

I’m a Scientist provides a unique way to study the ways children interact with scientists. We’d love to see more research being done using our data. If you think you, or someone you know, might be interested in analysing what hundreds of children want to know about science, just drop me a line at antony@gallomanor.com.

Read more on the findings in the full blog from the IOP: Online event gives insight into gender imbalances

And find out more about how Osmium Zone was set up: I’m a Scientist, not just for scientists

Posted on October 21, 2016 modantony in Evaluation, News | Leave a comment

Careers Zone

Report Cover

Careers Zone report, click to download

The Careers Zone is a new zone where students attending a careers fair at their school can talk online with a range of scientists and engineers, based all over the UK.

A lot of the time, school careers fairs are only able to accommodate businesses and organisations in the local area. We want to give students an opportunity to ask their questions to experts working in a multitude of areas, letting them see how diverse STEM careers can be. In turn, STEM professionals can pass on honest advice and information, showing students what it’s like to be a real engineer or scientist.

We selected a range of experts to take part, including a deep-sea researcher, a scientist for the NHS and a Jaguar Land Rover engineer. There were also engineers who had gotten into their professions through an apprenticeship. It was important for us to let students see the different options they have as they make decisions about their future careers. Experts could log in to the site from wherever they were for their scheduled Live Chat sessions, and then answer questions sent to them as they had time throughout the day (we even had an engineer take part in a chat from Mexico).

We set up on a long table with iPads and laptops for students to sit down at and use. We had two banners which showed some example questions, but often students came with a specific area of interest and asked us who the best expert was for them to talk with.

After two pilots, we’ve learnt it’s important for us to be on the same level as all the other stands at the fair, making approachable for students to ask who we are, and whether they can join in. We were also told that there was an announcement in the school newsletter about us being at the fair, and students had been preparing questions during tutorial times. This meant they came to the stand interested and prepared, making the chats lively and focussed. We’ve left the zone open so students can continue to log in, ask questions and leave comments.

Careers ZoneThe project clearly works at some level, but we are keen to pilot it in different schools. There are things we’ll have to change for next time, such as shortening the log in process, which can take valuable time away from students who only have 30 minutes or less at the fair. We’d also like to try a new way of showing students how to use the site. We’ve written up a report about our second pilot, including some examples of conversations that happened in the chat and ideas for next time, which you can read here.

Posted on October 6, 2016 modmichaela in Evaluation, News, Project News | Leave a comment

Osmium Zone – I’m a Scientist, not just for scientists?

UPDATE: Read about the Institute of Physics’ research into gender differences in online engagement here.

In all the zones we run, we aim to include a group of scientists that show how diverse STEM careers can be. At the start of the last school year our long-time collaborators, the Institute of Physics, asked us to take this idea a step further in the November 2015 Osmium Zone.

A zone usually comprises five people working at the cutting edge of scientific research. This time the IOP wanted to fund a zone with four people who had studied Physics at A-level or higher, and now worked outside of academia and research, plus one physics researcher. The kicker? Ideally, those four people would also be in jobs that made use of their Physics education.

Why? Studies show that children are not aware that studying science gives you transferable skills valuable in fields outside of traditional research. Making children more aware of this fact is a key recent recommendation from the ASPIRES project into children’s aspirations.

...And reply they did!

…And reply they did!

The first unknown: Would people outside the usual research audience be interested? Answer? Yes. A call went out on twitter, and very quickly we had a range of people interested in the zone. The final selection of the Osmium Zone consisted of a communications officer for the Royal Academy of Engineering with an undergraduate degree in physics, a diplomat at the British Embassy in Tokyo working in nuclear disposal,  a biomedical engineer, and a data analyst for a solar company. The ‘token scientist’ was a  postdoctoral researcher studying lasers.

So what happened? We were confident that both competitors and children would still find the event as engaging as ever, away from the usual science focus. This is illustrated every year in the sister I’m an Engineer project that covers the diverse world of engineering. The metrics for activity in the zone point to to this holding true, showing busy live chats and especially high numbers of page views for the final two contestants, Aaron and Natalie.

As usual, the students were keen to understand the choices and motivations of the different experts, asking questions like “Why did you choose this job?”. This allowed the group to talk about how studying science had led them to where they were, and how that knowledge was valuable to them now.

By letting the children discover for themselves the specific details of each person’s job, questions about these careers naturally followed:  Do you enjoy helping and working with Japanese companies and cities?What did you take (subjects wise) to be workin with solar power?.

Furthermore, students who completed a survey before and after the event indicated a slight increase in wanting a job that uses science skills and knowledge, although the sample size is too small to draw conclusions. We will complete a proper analysis on this, and other Science Capital related outcomes, after analysing data from multiple zones and events.

Safe to say, the zone saw heated competition.

So a zone including non-scientists works. The strength of the I’m a… format is that it harnesses the power of connecting students with real people, regardless of background. What’s next? We’d like to run more zones in the future that demonstrate the diversity of science-related careers to school students, and maybe even zones completely unrelated to science. Why not I’m a Poet, Get me out of here?

For now we’re trialing a Careers Zone with alumni from past events and it’s already been fascinating seeing some of the places former researchers now work. Since November, even the token scientist in Osmium Zone has moved to a non-academic role. Natalie now works for the Met Office, coordinating efforts to maximise the impact of research into climate change, and proving further that studying science can take you to interesting places. 


Read the Osmium Zone Report for more information about the zone

Read an interview with Keith Franklin about his experience as part of the zone

Posted on September 1, 2016 modantony in Evaluation, Event News, News | Leave a comment

Demand vs. Capacity — An update for June 2016

Any avid readers of our project blog — there must be at least one of you — will have noticed we’ve written a lot recently about our increasing demand for classes, and our over-subscription rates.

In January, we published some numbers. Since then we have run the March 2016 event, and finalised class places in the June 2016 event. Here are some updated numbers:

Graph of I'm a Scientist UK class requests by event in academic year — March 2013 to June 2016

I’m a Scientist UK class requests by event in academic year — March 2013 to June 2016

In January, we wrote about the decreasing popularity of the June events, with more teachers moving to November and March.

With the spectacular growth in November 2015, we speculated that teachers may be moving from March and June to earlier in the academic year. If that was true, then we would have expected a lower demand in March and June this year.

What we see is a steady increase in demand in the June events. The growth in March 2016 was perhaps less than may have been expected given that of previous years. This could support the idea that teachers are opting to take part in November instead of March. The narrative is not entirely clear though and we need to look into this a little more.

As was true in January, what is clear is the decreasing capacity for classes in all of the events. Though this does though coincide with an increased capacity in our other projects; with the demand for classes increasing we need to increase the capacity. We need more funding.

For the past year or so we have been charging international schools wishing to take part, generating a few hundred pounds. This June we began asking the same charge of independent schools. Overall, the response has been positive — which to be honest has been a pleasant surprise. There’s a separate post to come on that, but the fact that teachers are willing to pay shows they value the activity, and that this could be a valuable funding stream in the future.

Posted on May 23, 2016 Moderator - Josh in Evaluation, IAS Event, News, Project News | Leave a comment

On student activity and simplified registrations

TL;DR: We made the site easier for students to access and the proportion of our audience engaging increased.

Like any company, we have a set of Key Performance Indicators which we use to keep an eye on how well we’re doing. One of those KPIs (professionals use abbreviations) is the percentage of active students during an event, what we will call, %AS.

%AS shows the number of students who log in to the site, and go on to — at the very least — ask a question, write a line of text in a live chat, cast a vote, or leave a comment. Basically, it shows the proportion of our audience who are actively engaging with the activity.

In July 2014 Rosie posted a message on our project management app of choice, pointing out that the %AS for the previous events had been falling to the level it was at during the project pilot.

Graph of percentage of active students per IAS UK event. June 2011 to March 2016

Graph of percentage of active students per IAS UK event. June 2011 to March 2016

So, what did we do about it?

If you looked at the graph, you’ll see that we’ve already given the game away (but this is a one-graph-blog-post, and we’re not about to pad this out with multiple views of the same graph).

We started pre-registering students.

By visiting schools to observe students taking part, we saw that asking students to create their own accounts was taking way too long, was way too complicated, and largely, unnecessary.

We completely stripped down the process students go through to first get access to the site.

Previous and updated student access process

Previous and updated student access process

Previously, students would use an “access code” to get to a registration page, where they create a username and password, give us an email address, answer some other questions including some evaluation questions on their views of STEM. Now, students are given a generic username and password which gives them instant access to the site. From there, they can choose to go in and answer the evaluation questions, create a display name, and fill in their profile. But if they choose, they can get instant access to the live chats, to the question page, to scientists’ profiles.

The moral of the story then… By observing students use the site, we learnt that the registration process was too complicated. Pre-registering accounts for students does add a little more time and admin to the running of the event than not; but effort that pays off by making the site simpler to use and access for the students taking part.

Posted on May 23, 2016 Moderator - Josh in Evaluation, Event News, IAS Event, News, Project News | Leave a comment

Demand for classes

Column chart showing demand for classes outstripping supply

When we started I’m a Scientist we used our March events to allow teachers to test the event and for them to come back in June with more classes. That time after school exams when teachers looked for something to inspire kids was the ideal time.

Times have changed since then. March remains popular as teachers look for activities to coincide with British Science Week, but June is no longer the most popular time of year. We think this is due to increasing numbers of schools starting the next year’s timetable and curriculum after exams and that fallow period is no longer fallow.

November is where we saw some spectacular growth in 2015. We’re not sure what has driven this – we’ll investigate and we’ll rebalance when in the year we run zones.

The other clear narrative from this graph is that excluding the lower than expected level of requests in June 2014, demand for I’m a Scientist is increasing and outstripping demand. The past four events have been oversubscribed and March looks like going the same way as November 2015. There are some advantages in terms of zones being busy, but we need a balance.

Posted on January 12, 2016 ModShane in Evaluation | Leave a comment

November 2015 – Bigger than ever

We’ve just published the latest batch of zone reports for the recent November 2015 round of I’m a Scientist. You can find them here, or at the bottom of this post, but before you dive into the pretty pie charts and wicked wordles, we felt it would be useful to provide some context to keep in mind as you read:

We had an inkling that November’s eight zones were going to be big. As we reported beforehand, due to extremely high demand from teachers, it was the first time ever that we had to turn schools away from I’m a Scientist. We also had to limit the number of classes of those taking part to one or two per zone to fit as many schools in as possible. Without doing this we would have had to run double the number of zones to cover the demand.

So, after the hype, exactly how big were the zones in November? Here are some headline figures:

  • 481 students, on average, logged in to each zone. Compare this to the historical average of 353, and a target of at least 300 per zone.
  • Students submitted an average of 956 questions per zone and half of the zones saw well over 1000. The event average is 717 questions.
  • 20 live chats per zone was the average number in November. This is also usually the maximum number we allow.
The averages for the November 2015 event and the averages over three years of I'm a Scientist UK. '% of students active' is percentage of students who logged in and used ASK, CHAT or VOTE. Asterisks denote new record averages for an event.

The averages for the November 2015 event and the averages over three years of I’m a Scientist UK. ‘% of students active’ is percentage of students who logged in and used ASK, CHAT or VOTE. Asterisks denote new record averages for an event.

While the figures above are incredible in terms of numbers of students interacting with the site, high zone numbers aren’t everything. In fact, what’s best for both students and scientists is running smaller zones and more of them.

We still had great feedback from the students, teachers and scientists in November, but the bigger a zone gets, there are potential consequences. Scientists can only do so much in two weeks and with so many live chats happening, we did see slightly less scientists per chat in some zones. There’s also less chance for students to come up with their own ASK questions that haven’t been asked before, and then receive answers made just for them. The lower proportion of approved questions to submitted questions reflects this.

We don’t want these consequences. We want every student to have their own high quality, personalised interaction with a scientist. An increased number of smaller zones would make this easier for everyone.

The situation we’re in also means we have to choose which schools take part, and in an effort to reach those who lack regular STEM engagement, we’ve started prioritising places for Widening Participation and rural schools. Ideally, however, we want all schools who register for an event to get a place in a zone.

Thanks to our current funders, and the sterling efforts of the scientists, November was a success. But if we’re near the limit of zone size now, what’s next? Luckily, the high demand for I’m a Scientist represents a huge opportunity for organisations to reach the public. Being oversubscribed means we can show there are hundreds of schools ready to engage with hundreds of keen scientists. All that’s needed is the funding for the zones to let them get on with it, and we’re going to try our best to make that happen.

November 2015 Reports:

Posted on December 17, 2015 modantony in Evaluation, Event News, News | Leave a comment

3 words to describe a scientist

Survey results from a Year 6 student - Swanmead Community School

Survey results from a Year 6 student – Swanmead Community School

In March 2014 we started running zones for primary students only, increasing the number of primary school students who take part in I’m a Scientist. And we wanted to know more about how the event affects them.

This time, we drifted away from our usual online environment and sent printed surveys to teachers taking part in the Colour Zone and the Thulium Zone, the two primary only zones we ran in June 2015.

We also wanted to check if the response rate to printed surveys was higher than the response rate to digital ones, and it is. We got pre and post event results from roughly 40% of the students who took part in the two primary zones. Usually around 10% of students who take part in the events fill in both the pre and post event digital surveys. Analysis and interpretation of online surveys is quicker and more efficient that printed ones, but getting teachers involved in the evaluation process definitely made a difference.

We asked students which 3 words they would use to describe a scientist, before and after taking part. We got responses from over 300 students from 9 different schools.

  • The students used more that 230 different words to describe scientists. They used 188 different words before taking part and 127 after, maybe indicating that they are clearer about what a scientist is after taking part. Awesome, cooperative, inquisitive, life-saving… are just some examples of the words they used.
  • ‘Intelligent’, ‘clever’ and ‘smart’ were the most popular describers, although students used them less often after taking part – 46% of students used at least one of these words pre-event compared with 40% post-event.
  • Students described scientists as ‘awesome’, ‘epic’, and ‘cool’ twice as many times after taking part in I’m a Scientist– 9.7% of students used these words post-event compared with 4.4% pre-event.

We grouped words in synonym clusters, and we represented the relative pre to post-event difference on word use by gender:

new graphOnly words that were used at least 10 times are represented.

From the graph you can see that girls and boys equally use less negative stereotype words like ‘crazy’ after taking part.  Emotional descriptors – like ‘fun’ or ‘exciting’ – are mentioned more frequently after the event. You can also see that boys are more likely to use the word ‘fun’ after the event than girls. However, girls were more likely to use the word ‘awesome’ after taking part.

What do you think? Is there anything that particularly calls your attention? We asked a primary schools teacher and she told us:

I’m not surprised by the ‘awesome’ leap – I don’t think children really understand what is involved in the daily work of scientists until they interact on I’m a Scientist. My own students have been blown away by talk of live sheep spines, looking at volcanoes in space, etc.” – Tracy Tyrrell

If you are a primary school teacher and you want to evaluate your students’ attitudes to science before and after doing I’m a Scientist, you can download PDF version of them here:

If you are an academic (or, really anyone!) and you are keen to see and analyse the raw data of these printed surveys, please let us know in the comments and we’ll share them with you.

Please leave a comment, let’s continue the dialogue.

Special thanks to Dr Jessica Hamer, for her advice on the the printed surveys design.

Posted on November 2, 2015 modangela in Evaluation | Leave a comment

Which browsers and devices are schools using?

Every year (see 2014) we take all our data, and look at how schools use the site; looking at how visitors access the site and how much that has changed in the past couple of years.

We’ve made a lot of changes in recent events, especially when it comes to registering accounts and using the site on mobile devices. We now pre-register all teachers and students, giving them usernames and passwords, so they can start asking questions immediately.

We can assume that the student visitors give a fair reflection of the general school IT facilities and system capabilities.

The graphs show the student data represented by dashed lines.

Device

We saw last year, that visits from mobile and tablet devices were increasing. and no surprise they continue to grow, while desktop usage drops slightly. For student tablet users there is little difference in content visited when compared to student desktop users.

We’re currently right in the middle of an overhaul to make the site more responsive and mobile friendly- which should make the user experience a lot smoother for all users. Allowing students who use tablets and mobiles the same experience as desktop users.

Browser

If desktop usage is going down somewhat, it’s unsurprising that Internet Explorer is also sinking,  with Chrome taking a the lead and Safari jumping up (a lot of tablets and mobile devices being Apple products, where the default browser is Safari). The rise in other browsers will be down to mobile and tablet browsers using Android, Blackberry and Opera.

IE versions

And finally the bane of web designers and developers lives.. The old versions of Internet Explorer. A collective sigh of relief, IE6 has finally drifted out of favour (available since 2003!!!!), IE7 is on the out, IE8 is dropping, as is IE9, even IE10 is dropping.. But IE11 has taken a big leap. Understandably when a new version is released, it will lead to older versions not being used, but the jump in the graph is still quite dramatic, compared to last years.

Posted on October 30, 2015 modemily in Evaluation, School | Leave a comment

Research vs Evaluation

Science Learning+ is a significant funding scheme provided jointly between the Wellcome Trust and the National Science Foundation.

Learning can happen anywhere and at any time. Science Learning+ is an international initiative that aims to understand the power of informal learning experiences inside and outside of school.

The second aim of the scheme is to

“bridge the practice and research gap”

At a seminar in July aimed at providing an update on the Phase I project an interesting conversation developed about that gap between Science Communication practitioners and researchers.

I heard one speaker talk about practitioners wanting to know if a hypothetical red headline would give a 3% uplift in visitors. I responded on twitter:

Not all practioners agreed with me. Some felt each project would be unique enough to warrent a rewriting of expectations

Others simply disagreed and place efficacy as something for researchers:

In the end 140 characters felt underpowered.

For me research and evaluation are different, but very related.

I expect research to tell me if an approach to science communication works and how it works. I expect evaluation to tell me how well a project is working and how it can be improved. I would like evaluation to draw upon the research to extrapolate that particular  activities will lead to particular outcomes.

For example using I’m a Scientist:

The feedback we get from our participants is that connecting online with scientists improves their attitude to science, and to jobs in science. We seem to find the changes in attitude among girls is greater than it is for boys.

I want some research to tell me why those conversations are improving attitudes and if those changes are persistent. I want the research to be telling me how online activity compares to offline activity and why.

I want research to tell me what characteristics of engagement deliver the best and most persistent improvements in attitude and achievement.

Then I want my evaluation to examine our work against those characteristics and to suggest ways to improve them.

Research  = why something works

Evaluation = how well something works

What do you think?

Posted on August 27, 2015 ModShane in Evaluation, Science Engagement | 1 Comment

Harwell Open Day Zone Report

HRWL LEGO PosterOn Saturday July 11th the Harwell Research Campus opened its doors to the public. 10,000 people were expected to come and see some of the most complicated and cutting edge science equipment in the world. Hundreds of volunteers working with that equipment were on hand to speak with the public and show them what happens.

We created a Harwell Zone to allow visitors to text in questions. It was promoted using posters as above.

The logic was that with 10,000 visitors some would not get to ask their questions, or may think of it later, or possibly be too shy to ask in person. It was a family day.

Working with the over-worked (understatement klaxon just sounded) outreach team at Diamond and Harwell we set the zone up and promoted it to exhibitors.

It wasn’t a great success. A mere 14 questions were sent.

However in the interest of continual improvement let’s look at what worked, what didn’t and most importantly what would be do differently if given the chance.

What worked?

The technology did. Visitors were able to text a question, get a response to manage their expectations and a notification when someone answered the question. We also had an enthusiastic response from exhibitors who signed up. However the enthusiasm was far far greater amongst I’m a Scientist alumni. It seems that we didn’t really reach many new scientists. It was in the main scientists we already knew who happened to be already exhibiting.

What didn’t work?

We didn’t reach new scientists. The vast majority of scientists there were not aware that they could have taken part. This meant that the online offering was very patchy. As you wandered around the trained eye (mine) occasionally picked out a poster. The only exception was the RAL Cyrogenics Lab where Vicky Bayliss had printed out extra posters and placed them all around the lab.

We didn’t reach the public. There simply was not enough publicity.

What would we do next time?

  • An online Q&A offering needs to be an integral and supplementary part of the open day. Exhibitors should be opting out not in. Taking some questions online should be seen as part of the overall experience. We need to brief potential participants better.
  • The online experience needs to match the offline experience. At Harwell some participants were running a desk, others signed up the entire lab. It was disjointed. It would work at the lab level better.
  • We should pay more attention to the offline visitor flow. At Harwell many labs had a distinct flow from start to finish. We should have been making sure as people left they not only handed back their lanyards but took away a leaflet offering the chance to ask follow-up questions.
  • Use the online zone to supplement the offline experience. At Harwell visitors were invited to view the Diamond Light Synchotron but there were explainers in there. It was a prime spot to publicise the zone.
  • Programme notes. People tend to keep the programme notes with them throughout the day and on the way home. That’s the time to mop up any unanswered questions.
  • Make more of the online zone. We could take feedback, promote new open days, communicate campaigns, point to more resources. The point is that if someone asks a question online they are in effect asking to engage in conversation. Same as offline. The difference being that online you have the ability to invite the visitor to restart that conversation at any time in the future.

 

Posted on August 24, 2015 ModShane in Evaluation, News | Leave a comment

Engaging under-served audiences – talk at Bristol Zoo

Last week I was asked to speak about our experience of engaging under-served audiences. Here are my notes from which I spoke:

IAS is free online activity that connects school children and scientists. Kids go online, read scientist profiles, ask questions, take part in live chats and vote for the one they want to win.

Split into zones of 5 scientists and about 350 students. Mixture of general and themed zones. Been running since 2008 and reached nearly 70,000 students and over 1,000 scientists and engineers have taken part.

Scientists hear about it through colleagues, organisational emails and social media. They apply for the chance to take part. Tough selection process 1:4 for general zones.

Students hear about it through their teachers who sign up to take part. Over subscribed for I’m a Scientist and many teachers don’t get as many classes as they would like.

One way we do reach students who don’t normally get involved in science engagement is through product design.

Anecdotally, if a scientist visits a school, a class, 1:3 kids might stick their hand up to ask a question. With the online, pseudonymous nature of the activity we tend to get nearly 90% of the students actively participating – asking questions, chatting, commenting or voting. A side effect of this equality of voice is that not only do the quieter pupils get to ask their question but the other, louder, more confident students learn that their quieter peers do have something interesting to say. On a micro-scale this is reaching new audiences. And it is important. Not every engagement style works for everyone.

And it seems to suit some young people who don’t always get a say.

IAS is a spin off from a different project called I’m a Councillor, Get me out of here.

The inspiration for that project came out of conversations with council officers trying to interest young people in local democracy. They were doing things like inviting kids to shadow councillors for the day or they would invite students to visit the council chamber – and marvel at the majesty of our democratic overlords. The idea of an online activity was attractive to many, even if for some it just meant they weren’t going to need to clean up the graffiti left behind in the council chamber.

The why for us was that councils were trying to engage with mini-politicians, not mini-citizens. For me that was the more important democratic objective.

Over the 7 years that project ran some of the most memorable encounters were between the councillors and students from traditionally under-served groups.

I could talk about the pupil from the special school asking about the “futcha” for Bexhill or the feedback from the teacher at the PRU in Derbyshire who told us it was the first time that their pupils had ever been taken seriously by an authority figure. But it was the live chat between Nadia and a couple of Cardiff City Councillors that I remember best. The chat had been booked for 8pm by Steve a youth centre manager in Cardiff. It wasn’t the greatest of chats. Nadia was the only person online and was demanding a swimming pool for her and her mates. The two councillors patiently explained that Nadia should get together with friends and start a petition. After about 20 minutes of chat punctuated by strings of ****’s as Nadia’s choice of words got caught by our profanity filter, the chat was brought to a sudden end with Nadia typing, Nah can’t be bothered, bye bitches.

A disaster I thought.

5 minutes later Steve, the youth centre manager called me. That was brilliant he told me. Nadia had been excluded from every school in Cardiff and banned from every youth group. That night’s chat was the first time that Steve had ever seen Nadia engage with anything. Normally 1 in 5 words was a swear word but tonight in was only 1 in 20. He was ecstatic.

The point is that for some groups online communication is better than face-to-face and not just because of geography.

But having established that a project is good for certain groups it isn’t always as simple as that to get them involved.

We also look at the promotion aspect of reaching under-served audiences. Traditionally and anecdotally we’re told that science engagement projects do tend to reach the better resourced schools where kids often have plenty of privilege. Which means some schools are not taking part. But it isn’t so easy to identify them. Tied in is the issue for universities of widening participation as part of their fair access agreement required to charge £9,000 per year.

But what does widening participation mean? Apparently no two universities share the same definition (perhaps this has changed by now). We looked at a number of measures: IDACI, POLAR (participation in local area), and GCSE 5+ – when we looked at how the schools participating in IAS compared to the national profile against each of these criteria we were quite happy. We were getting a broad cross-section. And when compared to another large scale science engagement project we were performing well. But still not well enough.

There are flaws in the use of these measures. IDACI and POLAR give results based on the postcode of the school rather than it’s catchment area. The two can differ greatly. Secondly many of the schools in the poorest inner-city areas would also get greater funding through pupil premium and city-weighted education funding, perhaps allowing teachers more opportunities to bring in outside science engagement activities. Combine that with the fact that inner-city schools are also more likely to be within a short distance of a university and perhaps the inner-city schools aren’t so underserved.

So we still have a challenge on defining underserved.

We’re currently trying to work with other organisations like STEMNet to form a definition but in the meantime one of the factors we are looking at is distance. We’re looking to target schools from the poorer coastal areas in the UK.

But targeting is easier said than done.

We’ve conducted a few small experiments in recent years.

1. For one zone recently we sent 20 secondary schools in the most income deprived areas of the UK a teacher pack for the event inviting them to participate in the upcoming event. The idea was to make it as simple as possible for them to join in. Not one of them did. Not great.

2. More recently we posted a letter and some flyers to 200 primary schools in rural and coastal areas inviting them to apply to take part. The letter explained that Y5/6 was particularly important and that since they were in a remote area they might gain the most benefit. We’ve had 2 sign up. Not bad. OK

3. For a Food Zone in 2013 we looked at the schools signed up for the zone and worked out which would “qualify” as widening participation schools. There were about 5. Each teacher was called to make sure that they had received their packs and that they understood how the event worked. We looked after them, gave them special treatment. Every single one showed up online compared to a usual 2/3rds rate. It worked.

The learning we take from this is that you should look after the people already showing an interest. You get a better impact than simply going after more and more of your target. It might sound obvious, but for us we don’t have the time to call every school.
But we now realise that the time and money we might have spent sending stuff out to schools who hadn’t shown an interest, would be better spent on looking after the ones who had already shown some interest.

We’ll be repeating the specific targeting of schools in remote areas and giving special attention to those who respond. Thank you.

Posted on June 8, 2015 modshane in Evaluation, Science Engagement, Widening Participation | Leave a comment

March 2012 – March 2015 Evaluation Report

We were awarded a Wellcome Trust Society Award to run I’m a Scientist from March 2012 until June 2014. We later received a grant extension for November 2014 to June 2015. This report is all about our learnings in these last three years: from March 2012 until March 2015.

Our main learning points are:

WT report cover

Click here to download the full report

1. I’m a Scientist has gone from 30 zones per year in 2012, to 54 zones scheduled in the 2014/2015 school year.
2. Expanding zones to different audiences: primary school students and general public shows.
3. I’m a Scientist is a public engagement boost for scientists.
4. I’m a Scientist gets students enthused about science.
5. I’m a Scientist reaches a diverse set of students.
6. Teachers come back, but tricky to track.
7. Students ASK about cancer, animals, and life and CHAT about science, scientists and work.
8. Moving forwards, further adaptation to new technologies (such as tablets and smartphones) is important.

Click here to download the full report.

Posted on May 26, 2015 modangela in Evaluation | Leave a comment

Understanding Animal Research – Google Hangout

UARBefore our most recent June 2014 I’m a Scientist event we partnered with Understanding Animal Research to run a Google Hangout for scientists who work with animals.

Students often ask the scientists for their views on animal testing, and we want to help give scientists the confidence to be open and talk about working with animals in their research.

John Meredith, Education Manager at UAR, ran this session with six scientists who were taking part in I’m a Scientist the following week:

The A word: how to talk about animals in medical research 
This informal webinar presented by Understanding Animal Research will look at how and why we should talk about animals in medical research. It will cover current public attitudes to animal research, the facts and figures, the value of openness and how to answer the tricky questions or deal with confrontation. There will be plenty of time for questions and discussion during and after the presentation, which should last around an hour.

We asked the scientists to fill in a short survey after they’d taken part in I’m a Scientist, to assess whether the Hangout was useful, and whether we should offer it before every event.

Five scientists gave very positive feedback. In summary they found the Hangout useful and informative, but often didn’t use the training in I’m a Scientist as students didn’t ask about animal research, and the scientists didn’t want to bring it up. If we run it again we could make it more interactive, giving the scientists more practice in answering potentially difficult questions during the session.

Was the Google Hangout good?

  • All 5 scientists said yes

Did you use the training during the event?

  • 3 said no, because it didn’t come up, and they didn’t want to mention it unless they had to
  • 1 said yes, finding themselves using the training many times during the event
  • 1 said a little, but they didn’t get many questions on it

Did it make you more confident to talk about animal research?

  • All 5 scientists said yes
  • 1 of the scientists mentioned they were wary of talking about animal research in the live chats, in case time ran out and they couldn’t explain their work properly

Would you recommend it to others?

  • All 5 scientists said yes

How would you improve it?

  • Open the session up and ask for individuals input more
  • Links to resources that show that animal experimentation is not ‘animal cruelty’
  • Have more scientists who use animals in their work, to encourage more open discussion among peers

And a few other comments:

“Very helpful and answered a lot of the questions I had about discussing animal research with the public”

“I have contacted UAR and organised a school visit of my own!”

Posted on August 6, 2014 in Evaluation, IAS Event, News | Leave a comment

Wellcome Trust engaging science day

How to share and disseminate the learning from your project and evaluation.

On 14th May the Wellcome Trust held one of their twice-yearly Engaging Science days. They’re an opportunity for grantholders to meet with each other and Wellcome staff, to share learning and ideas. And to sample Wellcome’s renowned lunch offerings (side salad complete with edible flowers and quail eggs, anyone?).

Gallomanor were invited to speak on the panel of the session convened by Ben Johnson of Graphic Science, with the blurb “Once you have completed your project how do you widen its impact by ensuring others learn from what did (and didn’t) go right? An effective evaluation can also help with securing further funding and a wider roll-out of your project.” Other panelists were Manisha Lalloo from the Royal Academy of Engineering giving a funder’s perspective, and Becky Parker & Dave Colhurst from Simon Langton Grammar School.

Evaluating the learning from our projects and disseminating it (both within the office to colleagues and to the wider SciComm community) is important to us, so this is a summary of what I spoke about in the 5-10 min slot:

Capture

Angela sums up my talk excellently

  1. Be brave in budgeting. If you think an external evaluator will help, budget for it. Plan honestly for the amount of staff time your evaluation process will take. 
  2. Write up and talk about your findings, however small they seem. We all have to report back to funders, but making these reports readable will encourage others to read them. Start with a concise executive summary, use visual cues to break up text. Beyond this, share what you’ve learnt with colleagues and peers. We have an evaluation site where we post snippets of learning. Think about who you’re writing these for and target them – funders, participants (scientists, teachers, general public), or practitioners?
  3. Plan viable dissemination routes. Be innovative with evaluation and people will notice and ask you to speak and share learning at conferences. We’ve found this a brilliant way of publicising our projects, and it’s bought in new funding from people in the audience, such as the online discussion zone for the Ri Life Fantastic CHRISTMAS LECTURES.

And my top tip for dissemination when Ben sprung the question on us? Be short and visual in reports.

Posted on May 19, 2014 in Evaluation, News | Leave a comment

I’m a Scientist zones for primary students

We’ve had plenty of primary schools take part in I’m a Scientist zones in the past, nestled in among secondary schools. Feedback from primary teachers and students shows they get a lot out of taking part. Primary schools are looking for science enrichment activities too, and talking with real scientists is exciting at primary school as well as secondary. Scientists often wouldn’t know the students were Year 5 & 6, not Year 7 & 8, if we didn’t tell them.

caesium

This March we decided to run 2 primary school only zones, for Year 5 & 6 students. In part to avoid possible situations of primary students reading questions on non-primary-friendly topics asked by secondary students in their zone (think sex, drugs, rock’n’roll). Partly to open up I’m a Scientist to a wider group of UK schools and students. And importantly because one of the conclusions from the ASPIRES project about young people’s science and career aspirations, is that STEM education projects need to begin earlier, at primary school.

xenon

The 2 primary zones – Caesium Zone & Xenon Zone – were general zones, each with 5 scientists from a range of research areas, but avoiding any non-primary-appropriate topics (no IVF experts for instance).

What we learnt:

  1. There is demand for primary zones. We had too many teachers wanting places and had to turn some away. We’ll be running more primary zones in future events. 
  2. Primary zones aren’t that much different to secondary zones. They felt like just any other zone. We don’t need to change the format for primary students. The main difference we saw was that the questions were often more factual than conceptual, live chats were less disruptive than with older students, and students left lots of comments thanking scientists for their answers.
  3. Being in a primary zone didn’t seem to affect the scientists’ experience of taking part. When we offered them a place we mentioned it was in a primary zone but that shouldn’t change how they approach engaging with the students.
  4. Online safety is more of a worry for primary teachers and parents. We need to be clearer about how secure the site is and giving advice, such as students not using their first and last names in their username. This is true for secondary zones, too.
"Did you always like science when you where in primary school"

“Did you always like science when you were in primary school”

Posted on April 10, 2014 in Evaluation, News | Leave a comment

How does I’m a Scientist affect students’ attitudes to science?

We know anecdotally that participating in the I’m a Scientist event has a positive effect on students’ attitudes to science. Teachers tell us that their students were buzzing, and that they understood more about science. Students thank scientists in live chats for an interesting lesson. They tell us that it is “better than Facebook”.

But being data geeks that isn’t good enough. We wanted to know just how much we were affecting the students. Were we affecting them all by the same amount? Girls, boys, year groups? Did being more active in the event mean a greater change in attitude?

Questions we ask students

Thanks in part to the pilot work by Robin Longdin, then a SciComm masters student at UWE, we ask every student registering for I’m a Scientist a set of 4 questions:

  • How does school make you feel about science?Registration Questions
  • Are you planning to choose a science subject at the next stage of your education?
  • Do you think jobs involving science are interesting?
  • When you finish your education, how likely are you to look for a job that uses your science knowledge and skills?

We also ask the same questions towards the end of the event and ask for the username so we can match against their initial answers, their gender, year group, zone and activity levels within the event.

Measuring attitudinal change

The important thing for us is not the answers they give but the difference between the beginning and end of the event. We want to know if after two weeks of being exposed to our scientists they feel differently.

Each question had a five point answer scale. We intentionally wanted a neutral middle answer. We can’t expect students to all feel passionately about science. We then allocated a numerical value to each answer. 2 for the most positive, -2 for the most negative. We then simply subtract the starting answer value from the end of event value to give us a value for change in attitude for each of the four questions for each of the student who answered.

Do more active students see a greater change in attitude?

There is a lot of data generated and one of the challenges is to consolidate it into meaningful clusters. For example what do we mean by levels of activity? Some students are lively in live chats. Others ask a lot of questions. Some do both. We looked at the 3 main types of activity: ASK, CHAT, VOTE and scored each student between 0 and 3 depending on how active they were. We then totalled the individual scores to give each student a score between 0 and 9 overall. That is the X-axis below.

The first thing to note is that the overall trend is that the more activity on the site the more positive the change in attitude. Secondly the change in attitude to the final question: “When you finish your education, how likely are you to look for a job that uses your science knowledge and skills?” is much lower than the other questions, and in fact sometimes negative even for those students who made the most of our erstwhile role models.

We need to investigate this further. It could be a flawed question? What does it really mean to 13 and 14 year olds. Do school students consider a PhD studentship to be a job? Are half our role models not helping with that question?

UPDATE:
We went back to the source data and found to our embarrassment that we had made a mistake and we were comparing the answers to the final question to the answers to the 3rd question. D’Oh! The good news for us is that when you use the right data, the story looks even better. Students emerge from I’m a Scientist feeling much more positive about jobs in science for themselves.

The greatest effect is on girls and Year 10 & 11 students

This graph shows the average change in attitude for 6 groups by question asked. It clearly shows that the event has the greatest effect on girls and on students in Years 10 & 11.

This is based on a total of 853 valid responses from students across 3 events from June 2013 to March 2014 in the UK. The n for the groups in the second chart are:
All: n=853
Girls: n=475
Boys: n=333
Primary: n=90
Years 7-9: n=589
Year 10 & 11: n=130

They don’t all add up to 853. Sometimes it is a group of girls and boys who register. We also had 21 6th formers respond, but that is too small a sample to use. And sometimes the default “Please choose” answer remained. Yes there are disproprtionately more girls than boys, but yes, more girls than boys do participate.

Leave a comment and start a conversation!

Finally, we do this analysis for our own sake so that we can better understand our event, but we publish it for your sake, whoever you are. We want other practitioners to share in our knowledge. Please leave a comment to say “thanks”, or to ask for clarification or for more information.

Posted on April 9, 2014 ModShane in Evaluation, News | 4 Comments

March, June or November?

When teachers ask me what aged students they should bring on I’m a Scientist, my answer is often something along the lines of “well, it’s developed for Year 9 students but all ages get something different out of taking part, so take your pick”.

The good news is that our data supports this. When students register we ask them what year group they’re in. There are Year 5s. There are also Year 13s. Around two thirds of the 30,000+ students registered so far are KS3 (Years 7, 8 & 9).

The bad news? It’s not really bad news as such, but it looks like I should also be advising teachers on the time of year they should take part with different year groups.

March and June are great for pre-GCSE students. Teachers tell us they’re looking for something fun to do with students at the end of the year. They’re less good for GCSE and A Level students who have mocks, coursework and shock horror, actual exams to contend with.

November is slap bang in the middle of careers time for older students, and who better to speak to about possible careers in science than 5 practicing scientists online?

Different year groups take part at different times in the year

Different year groups take part at different times in the year

And just in case you’re wondering why there are a suspiciously high % of Year 5’s – that’s the registration form’s default option. So you might even expect it to be higher!

Posted on March 14, 2014 in Evaluation, News | Leave a comment

Girls vs Boys

I was asked recently what the gender split was for students taking part in I’m a Scientist. I replied that we’d never looked on the assumption that since teachers took their students online it would reflect the school population.

I’m never one to turn down the chance to crunch some numbers so I took a look. And was surprised. At registration we ask if the person registering is a Boy, Girl or A group of students (where more than one pupil is sharing a log-in). We have data going back to June 2012 and nearly 15,000 respondents to the question.

IAS Gender Split

Surprisingly it wasn’t a straight 50:50 split. So I then decided to look at it by year group.

IAS Gender by Year Group

What we saw was that year 9 & 10 were significantly skewed towards girls. And that bias continued through to 6th form.

Is that common for science outreach events?

Posted on March 11, 2014 ModShane in Evaluation | Leave a comment

I’m a Scientist so far in numbers

How many zones have we run? How many students engaging with how many scientists? Asking how many questions? What year group are they? Where in the UK are they?

The first slide of the I'm a Scientist summary so far...

The first slide of the I’m a Scientist summary so far…

These are some of the questions we get asked a lot, and until now haven’t had all the answers in one place to roll off the tip of our tongues. We’re pulling together some slides summarising I’m a Scientist to date and once they’re done we’ll post them up here.

In the mean time, I’ll be posting snippets that show just how far we’ve come since our first big Wellcome Trust grant in 2010.

To start us off (and to set the scene for the graphs and maps to come in future posts) here are some of the numbers so far, up to February 2014:

Zones: 118

  • 51 general science zones, 67 themed zones
  • Spread across 10 events, in March, June & November each year

Scientists: 590

Students: 36,225

  • % of students that actively took part: 84%
  • Average number of students per zone: 307
  • From 462 schools

Number of questions asked: 86,645

  • Number of questions approved: 38,836
  • Number of answers given: 84,890

Live chats: 1,616

  • Lines of live chat: 528,035

Visits to imascientist.org.uk: 1,592,016

  • Unique visitors: 1,306,978
  • Page views: 4,958,725
  • Average visit duration: 3:02 minutes
Posted on February 14, 2014 in Evaluation, News | Leave a comment