Our events generate huge amounts of interesting data, which we know contains all sorts of valuable insights. When there’s so much of it, it can take a while to work out how to make best use of it. One example of this is thematic analysis of the questions students post in ASK. Over the years of running the event we’ve developed a pretty good sense of the types of things students ask about and will always showcase great examples of questions. Converting that into a systematised, reportable analysis is more difficult, but can be done. So, we have developed a system for coding questions based on a set of themes that students often ask about, and after some trials in March, we’ve applied it to all zones in the June event (read the latest reports here).

At their broadest level, most student questions fall into one of three areas: questions about the science; questions about the scientist; questions about the I’m a… event. But we can look at this in more detail, considering elements of science capital (look out for a separate blog on this coming soon), and “how science works” (although this term has now been re-framed as “working scientifically” in the latest UK curriculum, we think it is still descriptive and useful for us). We’ve identified five top-level themes that we will report on. Each of which has one or more sub-themes which we are using to code the questions.

 

Science topics

covers facts, theories and knowledge about science. These can be related to the scientists work, or more general questions about science topics. Examples include…

When was the first epidemic discovered?

What animal has the worst venom and what does it do?

Can you give a robot human emotions?

How science works

looks at the process, motivations and ethics behind science. This could be finding out more about how decisions are made, why people chose science, or looking at how science fits into wider society. Examples include…

Is your work accurate and can be depended on?

Why do you think that a lot of people believe that psychology isn’t a science?

How did you come up with the idea of making MRI scans faster?

Career and education

includes the experiences of the scientists and aspirations of the students. Questions in this category build a greater understanding of real-life careers and education, including previous experience, current situation and future plans. Examples include…

What criteria did you use to choose your university?

Do you think you’ll ever leave the science job route for something different?

What is a PhD and how do you get one?

Personal

encompasses questions about scientists lives, knowledge and opinions outside of their work. Examples include….

Do you still have a social life?

From studying human relationships have you altered how you interact with people in day to day life?

What’s your opinion on the election?

Event/other

includes questions about the event and prize money, as well as a handful of questions that really don’t fit anywhere else. Examples include…

Do you think you can win I’m a Scientist?

Method

Using moderators and the event team each zone is coded, and then checked, so each set of questions have at least two run throughs. We use a coding guide as reference for consistency, and we are always building on our bank of reference questions for each category.

Students can ask anything they want, so it’s not always a clear-cut decision. Some questions overlap the different categories, but in general fit best within one of the reportable themes. There are some trickier questions that a few people will make a decision on together, and when we see a lot of similar questions we can agree how to code these. Although this won’t eliminate the ambiguity, it will help us to be consistent and transparent in how we code.

 

Results

Here’s how the questions distribute across all I’m a Scientist zones for the June 2017 events.

 

You can see the outcomes for individual zones in the latest I’m a Scientist reports and I’m an Engineer reports. We’re going to keep refining this coding to make it more consistent and the best reflection of our events it can be.

If you have any comments, thoughts, or would like to know more, please get in touch.

“I thought scientists just looked like they do in the film Flubber and experimented on aliens or weird stuff but when I found out you liked Taylor Swift I realised you are more down to earth and not like mad scientist :)” – Student, November 2015

I’m a Scientist is about connecting pupils with real scientists. Something we want to find out is what effect this interaction has on different groups of students. Recent research by the Institute of Physics has done just that.

The IOP has carried out an independent research project to gain an insight into the behaviour and attitudes of boys and girls who do I’m a Scientist. As part of their Improving Gender Balance project, the IOP funded two zones: Terbium Zone and Osmium Zone.

The researchers anonymously surveyed the students’ attitudes to science and scientists before and after the event. We also provided them with all the student’s interaction data from the sites. Every line of live chat, every question in ASK. Specifically the researchers wanted to look for gender differences in two areas:

1. How taking part in the event changed students’ attitudes to science and scientists, and
2. The types of questions that students asked the scientists.

The first results from analysis of this treasure trove are now online at the IOP blog.

Physicist, know thyself: After the event, girls in particular reduce the negative words they use to describe you.

Check out the post for their findings on the effects of taking part in I’m a Scientist, including:

• positive changes in word choice to describe scientists
• an increase in girls’ awareness of careers where Physics A-Level is useful
• girls’ improved confidence in talking to physicists

Girls want to see the scientists as real people

Click to see all the questions asked in Osmium Zone

The report also dissects what girls and boys want to know from the scientists. Interestingly, as well as asking more questions overall, girls seem to ask many more personal questions, asking more than boys about job satisfaction, the career goals and achievements of the scientists, and their motivation to work in science.

For us, the identified trends make sense. For example, a boisterous classroom culture can be a barrier for quieter children to engage with visitors. As Natasha at the IOP notes, ‘the online, anonymous nature of the live chats gives students more freedom to ask questions than a traditional careers talk or even a speed-networking-style careers event.’ Girls, as a group, seem to relish this freedom.

And we believe that for science to be appealing there’s no need to glam it up with explosions and ‘wonder’. The more students hear about the reality of science from people like them, the more positive they feel about it. This research backs up that view.

Research with us

If girls are a group that benefits from this increased exposure to scientists as real people, it’s likely other underrepresented groups do too.

I’m a Scientist provides a unique way to study the ways children interact with scientists. We’d love to see more research being done using our data. If you think you, or someone you know, might be interested in analysing what hundreds of children want to know about science, just drop me a line at antony@gallomanor.com.

Read more on the findings in the full blog from the IOP: Online event gives insight into gender imbalances

And find out more about how Osmium Zone was set up: I’m a Scientist, not just for scientists

On Saturday July 11th the Harwell Research Campus opened its doors to the public. 10,000 people were expected to come and see some of the most complicated and cutting edge science equipment in the world. Hundreds of volunteers working with that equipment were on hand to speak with the public and show them what happens.

We created a Harwell Zone to allow visitors to text in questions. It was promoted using posters as above.

The logic was that with 10,000 visitors some would not get to ask their questions, or may think of it later, or possibly be too shy to ask in person. It was a family day.

Working with the over-worked (understatement klaxon just sounded) outreach team at Diamond and Harwell we set the zone up and promoted it to exhibitors.

It wasn’t a great success. A mere 14 questions were sent.

However in the interest of continual improvement let’s look at what worked, what didn’t and most importantly what would be do differently if given the chance.

What worked?

The technology did. Visitors were able to text a question, get a response to manage their expectations and a notification when someone answered the question. We also had an enthusiastic response from exhibitors who signed up. However the enthusiasm was far far greater amongst I’m a Scientist alumni. It seems that we didn’t really reach many new scientists. It was in the main scientists we already knew who happened to be already exhibiting.

What didn’t work?

We didn’t reach new scientists. The vast majority of scientists there were not aware that they could have taken part. This meant that the online offering was very patchy. As you wandered around the trained eye (mine) occasionally picked out a poster. The only exception was the RAL Cyrogenics Lab where Vicky Bayliss had printed out extra posters and placed them all around the lab.

We didn’t reach the public. There simply was not enough publicity.

What would we do next time?

• An online Q&A offering needs to be an integral and supplementary part of the open day. Exhibitors should be opting out not in. Taking some questions online should be seen as part of the overall experience. We need to brief potential participants better.
• The online experience needs to match the offline experience. At Harwell some participants were running a desk, others signed up the entire lab. It was disjointed. It would work at the lab level better.
• We should pay more attention to the offline visitor flow. At Harwell many labs had a distinct flow from start to finish. We should have been making sure as people left they not only handed back their lanyards but took away a leaflet offering the chance to ask follow-up questions.
• Use the online zone to supplement the offline experience. At Harwell visitors were invited to view the Diamond Light Synchotron but there were explainers in there. It was a prime spot to publicise the zone.
• Programme notes. People tend to keep the programme notes with them throughout the day and on the way home. That’s the time to mop up any unanswered questions.
• Make more of the online zone. We could take feedback, promote new open days, communicate campaigns, point to more resources. The point is that if someone asks a question online they are in effect asking to engage in conversation. Same as offline. The difference being that online you have the ability to invite the visitor to restart that conversation at any time in the future.

How to share and disseminate the learning from your project and evaluation.

On 14th May the Wellcome Trust held one of their twice-yearly Engaging Science days. They’re an opportunity for grantholders to meet with each other and Wellcome staff, to share learning and ideas. And to sample Wellcome’s renowned lunch offerings (side salad complete with edible flowers and quail eggs, anyone?).

Gallomanor were invited to speak on the panel of the session convened by Ben Johnson of Graphic Science, with the blurb “Once you have completed your project how do you widen its impact by ensuring others learn from what did (and didn’t) go right? An effective evaluation can also help with securing further funding and a wider roll-out of your project.” Other panelists were Manisha Lalloo from the Royal Academy of Engineering giving a funder’s perspective, and Becky Parker & Dave Colhurst from Simon Langton Grammar School.

Evaluating the learning from our projects and disseminating it (both within the office to colleagues and to the wider SciComm community) is important to us, so this is a summary of what I spoke about in the 5-10 min slot:

Angela sums up my talk excellently

1. Be brave in budgeting. If you think an external evaluator will help, budget for it. Plan honestly for the amount of staff time your evaluation process will take. 
2. Write up and talk about your findings, however small they seem. We all have to report back to funders, but making these reports readable will encourage others to read them. Start with a concise executive summary, use visual cues to break up text. Beyond this, share what you’ve learnt with colleagues and peers. We have an evaluation site where we post snippets of learning. Think about who you’re writing these for and target them – funders, participants (scientists, teachers, general public), or practitioners?
3. Plan viable dissemination routes. Be innovative with evaluation and people will notice and ask you to speak and share learning at conferences. We’ve found this a brilliant way of publicising our projects, and it’s bought in new funding from people in the audience, such as the online discussion zone for the Ri Life Fantastic CHRISTMAS LECTURES.

And my top tip for dissemination when Ben sprung the question on us? Be short and visual in reports.

We’ve had plenty of primary schools take part in I’m a Scientist zones in the past, nestled in among secondary schools. Feedback from primary teachers and students shows they get a lot out of taking part. Primary schools are looking for science enrichment activities too, and talking with real scientists is exciting at primary school as well as secondary. Scientists often wouldn’t know the students were Year 5 & 6, not Year 7 & 8, if we didn’t tell them.

This March we decided to run 2 primary school only zones, for Year 5 & 6 students. In part to avoid possible situations of primary students reading questions on non-primary-friendly topics asked by secondary students in their zone (think sex, drugs, rock’n’roll). Partly to open up I’m a Scientist to a wider group of UK schools and students. And importantly because one of the conclusions from the ASPIRES project about young people’s science and career aspirations, is that STEM education projects need to begin earlier, at primary school.

The 2 primary zones – Caesium Zone & Xenon Zone – were general zones, each with 5 scientists from a range of research areas, but avoiding any non-primary-appropriate topics (no IVF experts for instance).

What we learnt:

1. There is demand for primary zones. We had too many teachers wanting places and had to turn some away. We’ll be running more primary zones in future events. 
2. Primary zones aren’t that much different to secondary zones. They felt like just any other zone. We don’t need to change the format for primary students. The main difference we saw was that the questions were often more factual than conceptual, live chats were less disruptive than with older students, and students left lots of comments thanking scientists for their answers.
3. Being in a primary zone didn’t seem to affect the scientists’ experience of taking part. When we offered them a place we mentioned it was in a primary zone but that shouldn’t change how they approach engaging with the students.
4. Online safety is more of a worry for primary teachers and parents. We need to be clearer about how secure the site is and giving advice, such as students not using their first and last names in their username. This is true for secondary zones, too.

“Did you always like science when you were in primary school”

We know anecdotally that participating in the I’m a Scientist event has a positive effect on students’ attitudes to science. Teachers tell us that their students were buzzing, and that they understood more about science. Students thank scientists in live chats for an interesting lesson. They tell us that it is “better than Facebook”.

But being data geeks that isn’t good enough. We wanted to know just how much we were affecting the students. Were we affecting them all by the same amount? Girls, boys, year groups? Did being more active in the event mean a greater change in attitude?

Questions we ask students

Thanks in part to the pilot work by Robin Longdin, then a SciComm masters student at UWE, we ask every student registering for I’m a Scientist a set of 4 questions:

• How does school make you feel about science?
• Are you planning to choose a science subject at the next stage of your education?
• Do you think jobs involving science are interesting?
• When you finish your education, how likely are you to look for a job that uses your science knowledge and skills?

We also ask the same questions towards the end of the event and ask for the username so we can match against their initial answers, their gender, year group, zone and activity levels within the event.

Measuring attitudinal change

The important thing for us is not the answers they give but the difference between the beginning and end of the event. We want to know if after two weeks of being exposed to our scientists they feel differently.

Each question had a five point answer scale. We intentionally wanted a neutral middle answer. We can’t expect students to all feel passionately about science. We then allocated a numerical value to each answer. 2 for the most positive, -2 for the most negative. We then simply subtract the starting answer value from the end of event value to give us a value for change in attitude for each of the four questions for each of the student who answered.

Do more active students see a greater change in attitude?

There is a lot of data generated and one of the challenges is to consolidate it into meaningful clusters. For example what do we mean by levels of activity? Some students are lively in live chats. Others ask a lot of questions. Some do both. We looked at the 3 main types of activity: ASK, CHAT, VOTE and scored each student between 0 and 3 depending on how active they were. We then totalled the individual scores to give each student a score between 0 and 9 overall. That is the X-axis below.

The first thing to note is that the overall trend is that the more activity on the site the more positive the change in attitude. Secondly the change in attitude to the final question: “When you finish your education, how likely are you to look for a job that uses your science knowledge and skills?” is much lower than the other questions, and in fact sometimes negative even for those students who made the most of our erstwhile role models.

We need to investigate this further. It could be a flawed question? What does it really mean to 13 and 14 year olds. Do school students consider a PhD studentship to be a job? Are half our role models not helping with that question?

UPDATE:
We went back to the source data and found to our embarrassment that we had made a mistake and we were comparing the answers to the final question to the answers to the 3rd question. D’Oh! The good news for us is that when you use the right data, the story looks even better. Students emerge from I’m a Scientist feeling much more positive about jobs in science for themselves.

The greatest effect is on girls and Year 10 & 11 students

This graph shows the average change in attitude for 6 groups by question asked. It clearly shows that the event has the greatest effect on girls and on students in Years 10 & 11.

This is based on a total of 853 valid responses from students across 3 events from June 2013 to March 2014 in the UK. The n for the groups in the second chart are:
All: n=853
Girls: n=475
Boys: n=333
Primary: n=90
Years 7-9: n=589
Year 10 & 11: n=130

They don’t all add up to 853. Sometimes it is a group of girls and boys who register. We also had 21 6th formers respond, but that is too small a sample to use. And sometimes the default “Please choose” answer remained. Yes there are disproprtionately more girls than boys, but yes, more girls than boys do participate.

Leave a comment and start a conversation!

Finally, we do this analysis for our own sake so that we can better understand our event, but we publish it for your sake, whoever you are. We want other practitioners to share in our knowledge. Please leave a comment to say “thanks”, or to ask for clarification or for more information.

When teachers ask me what aged students they should bring on I’m a Scientist, my answer is often something along the lines of “well, it’s developed for Year 9 students but all ages get something different out of taking part, so take your pick”.

The good news is that our data supports this. When students register we ask them what year group they’re in. There are Year 5s. There are also Year 13s. Around two thirds of the 30,000+ students registered so far are KS3 (Years 7, 8 & 9).

The bad news? It’s not really bad news as such, but it looks like I should also be advising teachers on the time of year they should take part with different year groups.

March and June are great for pre-GCSE students. Teachers tell us they’re looking for something fun to do with students at the end of the year. They’re less good for GCSE and A Level students who have mocks, coursework and shock horror, actual exams to contend with.

November is slap bang in the middle of careers time for older students, and who better to speak to about possible careers in science than 5 practicing scientists online?

Different year groups take part at different times in the year

And just in case you’re wondering why there are a suspiciously high % of Year 5’s – that’s the registration form’s default option. So you might even expect it to be higher!

How many zones have we run? How many students engaging with how many scientists? Asking how many questions? What year group are they? Where in the UK are they?

The first slide of the I’m a Scientist summary so far…

These are some of the questions we get asked a lot, and until now haven’t had all the answers in one place to roll off the tip of our tongues. We’re pulling together some slides summarising I’m a Scientist to date and once they’re done we’ll post them up here.

In the mean time, I’ll be posting snippets that show just how far we’ve come since our first big Wellcome Trust grant in 2010.

To start us off (and to set the scene for the graphs and maps to come in future posts) here are some of the numbers so far, up to February 2014:

Zones: 118

• 51 general science zones, 67 themed zones
• Spread across 10 events, in March, June & November each year

Scientists: 590

Students: 36,225

• % of students that actively took part: 84%
• Average number of students per zone: 307
• From 462 schools

Number of questions asked: 86,645

• Number of questions approved: 38,836
• Number of answers given: 84,890

Live chats: 1,616

• Lines of live chat: 528,035

Visits to imascientist.org.uk: 1,592,016

• Unique visitors: 1,306,978
• Page views: 4,958,725
• Average visit duration: 3:02 minutes

We’ve published all our individual zone reports from the June 2013 event. If you delve further back into the archives, you’ll find them for the zones we ran in March 2013 & November 2012 too. All the reports are tagged (funnily enough) under “Evaluation Reports“.

Each report sums up the activity in that zone. They provide some information for the scientists and teachers in that zone about how much the students and scientists interacted.

How many questions were asked and answered? What were the popular topics asked by students? What examples of great engagement stood out? What did students, scientists & teachers say about taking part? How does the winner plan to spend their £500 on communicating more science?

We email out each report to all the scientists and teachers in that zone. Scientists tell us the reports really help justify their participation to supervisors and colleagues. The detail tells them far more than just receiving a participation certificate in the post. The ‘loads’ of questions they answered turns out to be in the hundreds. The live chats went in a flash thanks to around 300 students bombarding them with questions over the 2 weeks. And the students really did ask about their research – volcanoes, bacteria & cancer for the zone below!

Take a look. If you’ve got any comments, such as what other info might be useful for scientists and teachers, we’d love to hear them.