Our events generate huge amounts of interesting data, which we know contains all sorts of valuable insights. When there’s so much of it, it can take a while to work out how to make best use of it. One example of this is thematic analysis of the questions students post in ASK. Over the years of running the event we’ve developed a pretty good sense of the types of things students ask about and will always showcase great examples of questions. Converting that into a systematised, reportable analysis is more difficult, but can be done. So, we have developed a system for coding questions based on a set of themes that students often ask about, and after some trials in March, we’ve applied it to all zones in the June event (read the latest reports here).At their broadest level, most student questions fall into one of three areas: questions about the science; questions about the scientist; questions about the I’m a… event. But we can look at this in more detail, considering elements of science capital (look out for a separate blog on this coming soon), and “how science works” (although this term has now been re-framed as “working scientifically” in the latest UK curriculum, we think it is still descriptive and useful for us). We’ve identified five top-level themes that we will report on. Each of which has one or more sub-themes which we are using to code the questions.
Science topics
covers facts, theories and knowledge about science. These can be related to the scientists work, or more general questions about science topics. Examples include…
When was the first epidemic discovered?
What animal has the worst venom and what does it do?
Can you give a robot human emotions?
How science workslooks at the process, motivations and ethics behind science. This could be finding out more about how decisions are made, why people chose science, or looking at how science fits into wider society. Examples include…
Is your work accurate and can be depended on?
Why do you think that a lot of people believe that psychology isn’t a science?
How did you come up with the idea of making MRI scans faster?
Career and educationincludes the experiences of the scientists and aspirations of the students. Questions in this category build a greater understanding of real-life careers and education, including previous experience, current situation and future plans. Examples include…
What criteria did you use to choose your university?
Do you think you’ll ever leave the science job route for something different?
What is a PhD and how do you get one?
Personalencompasses questions about scientists lives, knowledge and opinions outside of their work. Examples include….
Do you still have a social life?
From studying human relationships have you altered how you interact with people in day to day life?
What’s your opinion on the election?
Event/otherincludes questions about the event and prize money, as well as a handful of questions that really don’t fit anywhere else. Examples include…
Do you think you can win I’m a Scientist?
MethodUsing moderators and the event team each zone is coded, and then checked, so each set of questions have at least two run throughs. We use a coding guide as reference for consistency, and we are always building on our bank of reference questions for each category.
Students can ask anything they want, so it’s not always a clear-cut decision. Some questions overlap the different categories, but in general fit best within one of the reportable themes. There are some trickier questions that a few people will make a decision on together, and when we see a lot of similar questions we can agree how to code these. Although this won’t eliminate the ambiguity, it will help us to be consistent and transparent in how we code.
Results
Here’s how the questions distribute across all I’m a Scientist zones for the June 2017 events.
You can see the outcomes for individual zones in the latest I’m a Scientist reports and I’m an Engineer reports. We’re going to keep refining this coding to make it more consistent and the best reflection of our events it can be.
If you have any comments, thoughts, or would like to know more, please get in touch.
Category Archives: Evaluation
What do students ASK about?
Gender differences in online engagement
“I thought scientists just looked like they do in the film Flubber and experimented on aliens or weird stuff but when I found out you liked Taylor Swift I realised you are more down to earth and not like mad scientist :)” – Student, November 2015
I’m a Scientist is about connecting pupils with real scientists. Something we want to find out is what effect this interaction has on different groups of students. Recent research by the Institute of Physics has done just that.
The IOP has carried out an independent research project to gain an insight into the behaviour and attitudes of boys and girls who do I’m a Scientist. As part of their Improving Gender Balance project, the IOP funded two zones: Terbium Zone and Osmium Zone.
The researchers anonymously surveyed the students’ attitudes to science and scientists before and after the event. We also provided them with all the student’s interaction data from the sites. Every line of live chat, every question in ASK. Specifically the researchers wanted to look for gender differences in two areas:
- How taking part in the event changed students’ attitudes to science and scientists, and
- The types of questions that students asked the scientists.
The first results from analysis of this treasure trove are now online at the IOP blog.
Physicist, know thyself: After the event, girls in particular reduce the negative words they use to describe you.
Check out the post for their findings on the effects of taking part in I’m a Scientist, including:
- positive changes in word choice to describe scientists
- an increase in girls’ awareness of careers where Physics A-Level is useful
- girls’ improved confidence in talking to physicists
Girls want to see the scientists as real people
The report also dissects what girls and boys want to know from the scientists. Interestingly, as well as asking more questions overall, girls seem to ask many more personal questions, asking more than boys about job satisfaction, the career goals and achievements of the scientists, and their motivation to work in science.
For us, the identified trends make sense. For example, a boisterous classroom culture can be a barrier for quieter children to engage with visitors. As Natasha at the IOP notes, ‘the online, anonymous nature of the live chats gives students more freedom to ask questions than a traditional careers talk or even a speed-networking-style careers event.’ Girls, as a group, seem to relish this freedom.
And we believe that for science to be appealing there’s no need to glam it up with explosions and ‘wonder’. The more students hear about the reality of science from people like them, the more positive they feel about it. This research backs up that view.
Research with us
If girls are a group that benefits from this increased exposure to scientists as real people, it’s likely other underrepresented groups do too.
I’m a Scientist provides a unique way to study the ways children interact with scientists. We’d love to see more research being done using our data. If you think you, or someone you know, might be interested in analysing what hundreds of children want to know about science, just drop me a line at antony@gallomanor.com.
Read more on the findings in the full blog from the IOP: Online event gives insight into gender imbalances
And find out more about how Osmium Zone was set up: I’m a Scientist, not just for scientists
Careers Zone
Careers Zone report, click to download
The Careers Zone is a new zone where students attending a careers fair at their school can talk online with a range of scientists and engineers, based all over the UK.
A lot of the time, school careers fairs are only able to accommodate businesses and organisations in the local area. We want to give students an opportunity to ask their questions to experts working in a multitude of areas, letting them see how diverse STEM careers can be. In turn, STEM professionals can pass on honest advice and information, showing students what it’s like to be a real engineer or scientist.
We selected a range of experts to take part, including a deep-sea researcher, a scientist for the NHS and a Jaguar Land Rover engineer. There were also engineers who had gotten into their professions through an apprenticeship. It was important for us to let students see the different options they have as they make decisions about their future careers. Experts could log in to the site from wherever they were for their scheduled Live Chat sessions, and then answer questions sent to them as they had time throughout the day (we even had an engineer take part in a chat from Mexico).
We set up on a long table with iPads and laptops for students to sit down at and use. We had two banners which showed some example questions, but often students came with a specific area of interest and asked us who the best expert was for them to talk with.
After two pilots, we’ve learnt it’s important for us to be on the same level as all the other stands at the fair, making approachable for students to ask who we are, and whether they can join in. We were also told that there was an announcement in the school newsletter about us being at the fair, and students had been preparing questions during tutorial times. This meant they came to the stand interested and prepared, making the chats lively and focussed. We’ve left the zone open so students can continue to log in, ask questions and leave comments.
The project clearly works at some level, but we are keen to pilot it in different schools. There are things we’ll have to change for next time, such as shortening the log in process, which can take valuable time away from students who only have 30 minutes or less at the fair. We’d also like to try a new way of showing students how to use the site. We’ve written up a report about our second pilot, including some examples of conversations that happened in the chat and ideas for next time, which you can read here.
Osmium Zone – I’m a Scientist, not just for scientists?
UPDATE: Read about the Institute of Physics’ research into gender differences in online engagement here.
In all the zones we run, we aim to include a group of scientists that show how diverse STEM careers can be. At the start of the last school year our long-time collaborators, the Institute of Physics, asked us to take this idea a step further in the November 2015 Osmium Zone.
A zone usually comprises five people working at the cutting edge of scientific research. This time the IOP wanted to fund a zone with four people who had studied Physics at A-level or higher, and now worked outside of academia and research, plus one physics researcher. The kicker? Ideally, those four people would also be in jobs that made use of their Physics education.
Why? Studies show that children are not aware that studying science gives you transferable skills valuable in fields outside of traditional research. Making children more aware of this fact is a key recent recommendation from the ASPIRES project into children’s aspirations.
…And reply they did!
The first unknown: Would people outside the usual research audience be interested? Answer? Yes. A call went out on twitter, and very quickly we had a range of people interested in the zone. The final selection of the Osmium Zone consisted of a communications officer for the Royal Academy of Engineering with an undergraduate degree in physics, a diplomat at the British Embassy in Tokyo working in nuclear disposal, a biomedical engineer, and a data analyst for a solar company. The ‘token scientist’ was a postdoctoral researcher studying lasers.
So what happened? We were confident that both competitors and children would still find the event as engaging as ever, away from the usual science focus. This is illustrated every year in the sister I’m an Engineer project that covers the diverse world of engineering. The metrics for activity in the zone point to to this holding true, showing busy live chats and especially high numbers of page views for the final two contestants, Aaron and Natalie.
As usual, the students were keen to understand the choices and motivations of the different experts, asking questions like “Why did you choose this job?”. This allowed the group to talk about how studying science had led them to where they were, and how that knowledge was valuable to them now.
By letting the children discover for themselves the specific details of each person’s job, questions about these careers naturally followed: Do you enjoy helping and working with Japanese companies and cities?; What did you take (subjects wise) to be workin with solar power?.
Furthermore, students who completed a survey before and after the event indicated a slight increase in wanting a job that uses science skills and knowledge, although the sample size is too small to draw conclusions. We will complete a proper analysis on this, and other Science Capital related outcomes, after analysing data from multiple zones and events.
Safe to say, the zone saw heated competition.
So a zone including non-scientists works. The strength of the I’m a… format is that it harnesses the power of connecting students with real people, regardless of background. What’s next? We’d like to run more zones in the future that demonstrate the diversity of science-related careers to school students, and maybe even zones completely unrelated to science. Why not I’m a Poet, Get me out of here?
For now we’re trialing a Careers Zone with alumni from past events and it’s already been fascinating seeing some of the places former researchers now work. Since November, even the token scientist in Osmium Zone has moved to a non-academic role. Natalie now works for the Met Office, coordinating efforts to maximise the impact of research into climate change, and proving further that studying science can take you to interesting places.
Read the Osmium Zone Report for more information about the zone
Read an interview with Keith Franklin about his experience as part of the zone
Demand vs. Capacity — An update for June 2016
Any avid readers of our project blog — there must be at least one of you — will have noticed we’ve written a lot recently about our increasing demand for classes, and our over-subscription rates.
In January, we published some numbers. Since then we have run the March 2016 event, and finalised class places in the June 2016 event. Here are some updated numbers:
I’m a Scientist UK class requests by event in academic year — March 2013 to June 2016
In January, we wrote about the decreasing popularity of the June events, with more teachers moving to November and March.
With the spectacular growth in November 2015, we speculated that teachers may be moving from March and June to earlier in the academic year. If that was true, then we would have expected a lower demand in March and June this year.
What we see is a steady increase in demand in the June events. The growth in March 2016 was perhaps less than may have been expected given that of previous years. This could support the idea that teachers are opting to take part in November instead of March. The narrative is not entirely clear though and we need to look into this a little more.
As was true in January, what is clear is the decreasing capacity for classes in all of the events. Though this does though coincide with an increased capacity in our other projects; with the demand for classes increasing we need to increase the capacity. We need more funding.
For the past year or so we have been charging international schools wishing to take part, generating a few hundred pounds. This June we began asking the same charge of independent schools. Overall, the response has been positive — which to be honest has been a pleasant surprise. There’s a separate post to come on that, but the fact that teachers are willing to pay shows they value the activity, and that this could be a valuable funding stream in the future.
Moderator - Josh in Capacity, Evaluation, IAS Event, News, Project News |
Leave a comment
On student activity and simplified registrations
TL;DR: We made the site easier for students to access and the proportion of our audience engaging increased.
Like any company, we have a set of Key Performance Indicators which we use to keep an eye on how well we’re doing. One of those KPIs (professionals use abbreviations) is the percentage of active students during an event, what we will call, %AS.
%AS shows the number of students who log in to the site, and go on to — at the very least — ask a question, write a line of text in a live chat, cast a vote, or leave a comment. Basically, it shows the proportion of our audience who are actively engaging with the activity.
In July 2014 Rosie posted a message on our project management app of choice, pointing out that the %AS for the previous events had been falling to the level it was at during the project pilot.
Graph of percentage of active students per IAS UK event. June 2011 to March 2016
So, what did we do about it?
If you looked at the graph, you’ll see that we’ve already given the game away (but this is a one-graph-blog-post, and we’re not about to pad this out with multiple views of the same graph).
We started pre-registering students.
By visiting schools to observe students taking part, we saw that asking students to create their own accounts was taking way too long, was way too complicated, and largely, unnecessary.
We completely stripped down the process students go through to first get access to the site.
Previous and updated student access process
Previously, students would use an “access code” to get to a registration page, where they create a username and password, give us an email address, answer some other questions including some evaluation questions on their views of STEM. Now, students are given a generic username and password which gives them instant access to the site. From there, they can choose to go in and answer the evaluation questions, create a display name, and fill in their profile. But if they choose, they can get instant access to the live chats, to the question page, to scientists’ profiles.
The moral of the story then… By observing students use the site, we learnt that the registration process was too complicated. Pre-registering accounts for students does add a little more time and admin to the running of the event than not; but effort that pays off by making the site simpler to use and access for the students taking part.
Moderator - Josh in Evaluation, Event News, IAS Event, News, Project News |
Leave a comment
Demand for classes
When we started I’m a Scientist we used our March events to allow teachers to test the event and for them to come back in June with more classes. That time after school exams when teachers looked for something to inspire kids was the ideal time.
Times have changed since then. March remains popular as teachers look for activities to coincide with British Science Week, but June is no longer the most popular time of year. We think this is due to increasing numbers of schools starting the next year’s timetable and curriculum after exams and that fallow period is no longer fallow.
November is where we saw some spectacular growth in 2015. We’re not sure what has driven this – we’ll investigate and we’ll rebalance when in the year we run zones.
The other clear narrative from this graph is that excluding the lower than expected level of requests in June 2014, demand for I’m a Scientist is increasing and outstripping demand. The past four events have been oversubscribed and March looks like going the same way as November 2015. There are some advantages in terms of zones being busy, but we need a balance.
November 2015 – Bigger than ever
We’ve just published the latest batch of zone reports for the recent November 2015 round of I’m a Scientist. You can find them here, or at the bottom of this post, but before you dive into the pretty pie charts and wicked wordles, we felt it would be useful to provide some context to keep in mind as you read:
We had an inkling that November’s eight zones were going to be big. As we reported beforehand, due to extremely high demand from teachers, it was the first time ever that we had to turn schools away from I’m a Scientist. We also had to limit the number of classes of those taking part to one or two per zone to fit as many schools in as possible. Without doing this we would have had to run double the number of zones to cover the demand.
So, after the hype, exactly how big were the zones in November? Here are some headline figures:
- 481 students, on average, logged in to each zone. Compare this to the historical average of 353, and a target of at least 300 per zone.
- Students submitted an average of 956 questions per zone and half of the zones saw well over 1000. The event average is 717 questions.
- 20 live chats per zone was the average number in November. This is also usually the maximum number we allow.
The averages for the November 2015 event and the averages over three years of I’m a Scientist UK. ‘% of students active’ is percentage of students who logged in and used ASK, CHAT or VOTE. Asterisks denote new record averages for an event.
While the figures above are incredible in terms of numbers of students interacting with the site, high zone numbers aren’t everything. In fact, what’s best for both students and scientists is running smaller zones and more of them.
We still had great feedback from the students, teachers and scientists in November, but the bigger a zone gets, there are potential consequences. Scientists can only do so much in two weeks and with so many live chats happening, we did see slightly less scientists per chat in some zones. There’s also less chance for students to come up with their own ASK questions that haven’t been asked before, and then receive answers made just for them. The lower proportion of approved questions to submitted questions reflects this.
We don’t want these consequences. We want every student to have their own high quality, personalised interaction with a scientist. An increased number of smaller zones would make this easier for everyone.
The situation we’re in also means we have to choose which schools take part, and in an effort to reach those who lack regular STEM engagement, we’ve started prioritising places for Widening Participation and rural schools. Ideally, however, we want all schools who register for an event to get a place in a zone.
Thanks to our current funders, and the sterling efforts of the scientists, November was a success. But if we’re near the limit of zone size now, what’s next? Luckily, the high demand for I’m a Scientist represents a huge opportunity for organisations to reach the public. Being oversubscribed means we can show there are hundreds of schools ready to engage with hundreds of keen scientists. All that’s needed is the funding for the zones to let them get on with it, and we’re going to try our best to make that happen.
November 2015 Reports:
3 words to describe a scientist
Survey results from a Year 6 student – Swanmead Community School
In March 2014 we started running zones for primary students only, increasing the number of primary school students who take part in I’m a Scientist. And we wanted to know more about how the event affects them.
This time, we drifted away from our usual online environment and sent printed surveys to teachers taking part in the Colour Zone and the Thulium Zone, the two primary only zones we ran in June 2015.
We also wanted to check if the response rate to printed surveys was higher than the response rate to digital ones, and it is. We got pre and post event results from roughly 40% of the students who took part in the two primary zones. Usually around 10% of students who take part in the events fill in both the pre and post event digital surveys. Analysis and interpretation of online surveys is quicker and more efficient that printed ones, but getting teachers involved in the evaluation process definitely made a difference.
We asked students which 3 words they would use to describe a scientist, before and after taking part. We got responses from over 300 students from 9 different schools.
- The students used more that 230 different words to describe scientists. They used 188 different words before taking part and 127 after, maybe indicating that they are clearer about what a scientist is after taking part. Awesome, cooperative, inquisitive, life-saving… are just some examples of the words they used.
- ‘Intelligent’, ‘clever’ and ‘smart’ were the most popular describers, although students used them less often after taking part – 46% of students used at least one of these words pre-event compared with 40% post-event.
- Students described scientists as ‘awesome’, ‘epic’, and ‘cool’ twice as many times after taking part in I’m a Scientist– 9.7% of students used these words post-event compared with 4.4% pre-event.
We grouped words in synonym clusters, and we represented the relative pre to post-event difference on word use by gender:
Only words that were used at least 10 times are represented.
From the graph you can see that girls and boys equally use less negative stereotype words like ‘crazy’ after taking part. Emotional descriptors – like ‘fun’ or ‘exciting’ – are mentioned more frequently after the event. You can also see that boys are more likely to use the word ‘fun’ after the event than girls. However, girls were more likely to use the word ‘awesome’ after taking part.
What do you think? Is there anything that particularly calls your attention? We asked a primary schools teacher and she told us:
I’m not surprised by the ‘awesome’ leap – I don’t think children really understand what is involved in the daily work of scientists until they interact on I’m a Scientist. My own students have been blown away by talk of live sheep spines, looking at volcanoes in space, etc.” – Tracy Tyrrell
If you are a primary school teacher and you want to evaluate your students’ attitudes to science before and after doing I’m a Scientist, you can download PDF version of them here:
If you are an academic (or, really anyone!) and you are keen to see and analyse the raw data of these printed surveys, please let us know in the comments and we’ll share them with you.
Please leave a comment, let’s continue the dialogue.
Special thanks to Dr Jessica Hamer, for her advice on the the printed surveys design.
Which browsers and devices are schools using?
Every year (see 2014) we take all our data, and look at how schools use the site; looking at how visitors access the site and how much that has changed in the past couple of years.
We’ve made a lot of changes in recent events, especially when it comes to registering accounts and using the site on mobile devices. We now pre-register all teachers and students, giving them usernames and passwords, so they can start asking questions immediately.
We can assume that the student visitors give a fair reflection of the general school IT facilities and system capabilities.
The graphs show the student data represented by dashed lines.
We saw last year, that visits from mobile and tablet devices were increasing. and no surprise they continue to grow, while desktop usage drops slightly. For student tablet users there is little difference in content visited when compared to student desktop users.
We’re currently right in the middle of an overhaul to make the site more responsive and mobile friendly- which should make the user experience a lot smoother for all users. Allowing students who use tablets and mobiles the same experience as desktop users.
If desktop usage is going down somewhat, it’s unsurprising that Internet Explorer is also sinking, with Chrome taking a the lead and Safari jumping up (a lot of tablets and mobile devices being Apple products, where the default browser is Safari). The rise in other browsers will be down to mobile and tablet browsers using Android, Blackberry and Opera.
And finally the bane of web designers and developers lives.. The old versions of Internet Explorer. A collective sigh of relief, IE6 has finally drifted out of favour (available since 2003!!!!), IE7 is on the out, IE8 is dropping, as is IE9, even IE10 is dropping.. But IE11 has taken a big leap. Understandably when a new version is released, it will lead to older versions not being used, but the jump in the graph is still quite dramatic, compared to last years.
Research vs Evaluation
Science Learning+ is a significant funding scheme provided jointly between the Wellcome Trust and the National Science Foundation.
Learning can happen anywhere and at any time. Science Learning+ is an international initiative that aims to understand the power of informal learning experiences inside and outside of school.
The second aim of the scheme is to
“bridge the practice and research gap”
At a seminar in July aimed at providing an update on the Phase I project an interesting conversation developed about that gap between Science Communication practitioners and researchers.
I heard one speaker talk about practitioners wanting to know if a hypothetical red headline would give a 3% uplift in visitors. I responded on twitter:
Disagree that practitioners want efficacy. I use eval. for that. I want research to tell me if the activity provides good outcomes #slplus
— Shane McCracken (@ShaneMcC) July 27, 2015
Not all practioners agreed with me. Some felt each project would be unique enough to warrent a rewriting of expectations
@ShaneMcC @arlloyd @LauraFoggRogers I'm not sure it has b/c each time res'ers & pract'ers collab it should be unique.
— Helen Featherstone (@HFeatherstone) July 28, 2015
Others simply disagreed and place efficacy as something for researchers:
@ShaneMcC I also want research to help me with process (e.g. relevant psychological mechanisms) – more front-end than summative #slplus
— Andy Lloyd (@arlloyd) July 27, 2015
In the end 140 characters felt underpowered.
For me research and evaluation are different, but very related.
I expect research to tell me if an approach to science communication works and how it works. I expect evaluation to tell me how well a project is working and how it can be improved. I would like evaluation to draw upon the research to extrapolate that particular activities will lead to particular outcomes.
For example using I’m a Scientist:
The feedback we get from our participants is that connecting online with scientists improves their attitude to science, and to jobs in science. We seem to find the changes in attitude among girls is greater than it is for boys.
I want some research to tell me why those conversations are improving attitudes and if those changes are persistent. I want the research to be telling me how online activity compares to offline activity and why.
I want research to tell me what characteristics of engagement deliver the best and most persistent improvements in attitude and achievement.
Then I want my evaluation to examine our work against those characteristics and to suggest ways to improve them.
Research = why something works
Evaluation = how well something works
What do you think?
Harwell Open Day Zone Report
On Saturday July 11th the Harwell Research Campus opened its doors to the public. 10,000 people were expected to come and see some of the most complicated and cutting edge science equipment in the world. Hundreds of volunteers working with that equipment were on hand to speak with the public and show them what happens.
We created a Harwell Zone to allow visitors to text in questions. It was promoted using posters as above.
The logic was that with 10,000 visitors some would not get to ask their questions, or may think of it later, or possibly be too shy to ask in person. It was a family day.
Working with the over-worked (understatement klaxon just sounded) outreach team at Diamond and Harwell we set the zone up and promoted it to exhibitors.
It wasn’t a great success. A mere 14 questions were sent.
However in the interest of continual improvement let’s look at what worked, what didn’t and most importantly what would be do differently if given the chance.
What worked?
The technology did. Visitors were able to text a question, get a response to manage their expectations and a notification when someone answered the question. We also had an enthusiastic response from exhibitors who signed up. However the enthusiasm was far far greater amongst I’m a Scientist alumni. It seems that we didn’t really reach many new scientists. It was in the main scientists we already knew who happened to be already exhibiting.
What didn’t work?
We didn’t reach new scientists. The vast majority of scientists there were not aware that they could have taken part. This meant that the online offering was very patchy. As you wandered around the trained eye (mine) occasionally picked out a poster. The only exception was the RAL Cyrogenics Lab where Vicky Bayliss had printed out extra posters and placed them all around the lab.
We didn’t reach the public. There simply was not enough publicity.
What would we do next time?
- An online Q&A offering needs to be an integral and supplementary part of the open day. Exhibitors should be opting out not in. Taking some questions online should be seen as part of the overall experience. We need to brief potential participants better.
- The online experience needs to match the offline experience. At Harwell some participants were running a desk, others signed up the entire lab. It was disjointed. It would work at the lab level better.
- We should pay more attention to the offline visitor flow. At Harwell many labs had a distinct flow from start to finish. We should have been making sure as people left they not only handed back their lanyards but took away a leaflet offering the chance to ask follow-up questions.
- Use the online zone to supplement the offline experience. At Harwell visitors were invited to view the Diamond Light Synchotron but there were explainers in there. It was a prime spot to publicise the zone.
- Programme notes. People tend to keep the programme notes with them throughout the day and on the way home. That’s the time to mop up any unanswered questions.
- Make more of the online zone. We could take feedback, promote new open days, communicate campaigns, point to more resources. The point is that if someone asks a question online they are in effect asking to engage in conversation. Same as offline. The difference being that online you have the ability to invite the visitor to restart that conversation at any time in the future.
Engaging under-served audiences – talk at Bristol Zoo
Last week I was asked to speak about our experience of engaging under-served audiences. Here are my notes from which I spoke:
IAS is free online activity that connects school children and scientists. Kids go online, read scientist profiles, ask questions, take part in live chats and vote for the one they want to win.
Split into zones of 5 scientists and about 350 students. Mixture of general and themed zones. Been running since 2008 and reached nearly 70,000 students and over 1,000 scientists and engineers have taken part.
Scientists hear about it through colleagues, organisational emails and social media. They apply for the chance to take part. Tough selection process 1:4 for general zones.
Students hear about it through their teachers who sign up to take part. Over subscribed for I’m a Scientist and many teachers don’t get as many classes as they would like.
One way we do reach students who don’t normally get involved in science engagement is through product design.
Anecdotally, if a scientist visits a school, a class, 1:3 kids might stick their hand up to ask a question. With the online, pseudonymous nature of the activity we tend to get nearly 90% of the students actively participating – asking questions, chatting, commenting or voting. A side effect of this equality of voice is that not only do the quieter pupils get to ask their question but the other, louder, more confident students learn that their quieter peers do have something interesting to say. On a micro-scale this is reaching new audiences. And it is important. Not every engagement style works for everyone.
And it seems to suit some young people who don’t always get a say.
IAS is a spin off from a different project called I’m a Councillor, Get me out of here.
The inspiration for that project came out of conversations with council officers trying to interest young people in local democracy. They were doing things like inviting kids to shadow councillors for the day or they would invite students to visit the council chamber – and marvel at the majesty of our democratic overlords. The idea of an online activity was attractive to many, even if for some it just meant they weren’t going to need to clean up the graffiti left behind in the council chamber.
The why for us was that councils were trying to engage with mini-politicians, not mini-citizens. For me that was the more important democratic objective.
Over the 7 years that project ran some of the most memorable encounters were between the councillors and students from traditionally under-served groups.
I could talk about the pupil from the special school asking about the “futcha” for Bexhill or the feedback from the teacher at the PRU in Derbyshire who told us it was the first time that their pupils had ever been taken seriously by an authority figure. But it was the live chat between Nadia and a couple of Cardiff City Councillors that I remember best. The chat had been booked for 8pm by Steve a youth centre manager in Cardiff. It wasn’t the greatest of chats. Nadia was the only person online and was demanding a swimming pool for her and her mates. The two councillors patiently explained that Nadia should get together with friends and start a petition. After about 20 minutes of chat punctuated by strings of ****’s as Nadia’s choice of words got caught by our profanity filter, the chat was brought to a sudden end with Nadia typing, Nah can’t be bothered, bye bitches.
A disaster I thought.
5 minutes later Steve, the youth centre manager called me. That was brilliant he told me. Nadia had been excluded from every school in Cardiff and banned from every youth group. That night’s chat was the first time that Steve had ever seen Nadia engage with anything. Normally 1 in 5 words was a swear word but tonight in was only 1 in 20. He was ecstatic.
The point is that for some groups online communication is better than face-to-face and not just because of geography.
But having established that a project is good for certain groups it isn’t always as simple as that to get them involved.
We also look at the promotion aspect of reaching under-served audiences. Traditionally and anecdotally we’re told that science engagement projects do tend to reach the better resourced schools where kids often have plenty of privilege. Which means some schools are not taking part. But it isn’t so easy to identify them. Tied in is the issue for universities of widening participation as part of their fair access agreement required to charge £9,000 per year.
But what does widening participation mean? Apparently no two universities share the same definition (perhaps this has changed by now). We looked at a number of measures: IDACI, POLAR (participation in local area), and GCSE 5+ – when we looked at how the schools participating in IAS compared to the national profile against each of these criteria we were quite happy. We were getting a broad cross-section. And when compared to another large scale science engagement project we were performing well. But still not well enough.
There are flaws in the use of these measures. IDACI and POLAR give results based on the postcode of the school rather than it’s catchment area. The two can differ greatly. Secondly many of the schools in the poorest inner-city areas would also get greater funding through pupil premium and city-weighted education funding, perhaps allowing teachers more opportunities to bring in outside science engagement activities. Combine that with the fact that inner-city schools are also more likely to be within a short distance of a university and perhaps the inner-city schools aren’t so underserved.
So we still have a challenge on defining underserved.
We’re currently trying to work with other organisations like STEMNet to form a definition but in the meantime one of the factors we are looking at is distance. We’re looking to target schools from the poorer coastal areas in the UK.
But targeting is easier said than done.
We’ve conducted a few small experiments in recent years.
1. For one zone recently we sent 20 secondary schools in the most income deprived areas of the UK a teacher pack for the event inviting them to participate in the upcoming event. The idea was to make it as simple as possible for them to join in. Not one of them did. Not great.
2. More recently we posted a letter and some flyers to 200 primary schools in rural and coastal areas inviting them to apply to take part. The letter explained that Y5/6 was particularly important and that since they were in a remote area they might gain the most benefit. We’ve had 2 sign up. Not bad. OK
3. For a Food Zone in 2013 we looked at the schools signed up for the zone and worked out which would “qualify” as widening participation schools. There were about 5. Each teacher was called to make sure that they had received their packs and that they understood how the event worked. We looked after them, gave them special treatment. Every single one showed up online compared to a usual 2/3rds rate. It worked.
The learning we take from this is that you should look after the people already showing an interest. You get a better impact than simply going after more and more of your target. It might sound obvious, but for us we don’t have the time to call every school.
But we now realise that the time and money we might have spent sending stuff out to schools who hadn’t shown an interest, would be better spent on looking after the ones who had already shown some interest.We’ll be repeating the specific targeting of schools in remote areas and giving special attention to those who respond. Thank you.
modshane in Evaluation, Science Engagement, Widening Participation |
Leave a comment
March 2012 – March 2015 Evaluation Report
We were awarded a Wellcome Trust Society Award to run I’m a Scientist from March 2012 until June 2014. We later received a grant extension for November 2014 to June 2015. This report is all about our learnings in these last three years: from March 2012 until March 2015.
Our main learning points are:
Click here to download the full report
1. I’m a Scientist has gone from 30 zones per year in 2012, to 54 zones scheduled in the 2014/2015 school year.
2. Expanding zones to different audiences: primary school students and general public shows.
3. I’m a Scientist is a public engagement boost for scientists.
4. I’m a Scientist gets students enthused about science.
5. I’m a Scientist reaches a diverse set of students.
6. Teachers come back, but tricky to track.
7. Students ASK about cancer, animals, and life and CHAT about science, scientists and work.
8. Moving forwards, further adaptation to new technologies (such as tablets and smartphones) is important.
Click here to download the full report.
Understanding Animal Research – Google Hangout
Before our most recent June 2014 I’m a Scientist event we partnered with Understanding Animal Research to run a Google Hangout for scientists who work with animals.
Students often ask the scientists for their views on animal testing, and we want to help give scientists the confidence to be open and talk about working with animals in their research.
John Meredith, Education Manager at UAR, ran this session with six scientists who were taking part in I’m a Scientist the following week:
The A word: how to talk about animals in medical research
This informal webinar presented by Understanding Animal Research will look at how and why we should talk about animals in medical research. It will cover current public attitudes to animal research, the facts and figures, the value of openness and how to answer the tricky questions or deal with confrontation. There will be plenty of time for questions and discussion during and after the presentation, which should last around an hour.
We asked the scientists to fill in a short survey after they’d taken part in I’m a Scientist, to assess whether the Hangout was useful, and whether we should offer it before every event.
Five scientists gave very positive feedback. In summary they found the Hangout useful and informative, but often didn’t use the training in I’m a Scientist as students didn’t ask about animal research, and the scientists didn’t want to bring it up. If we run it again we could make it more interactive, giving the scientists more practice in answering potentially difficult questions during the session.
Was the Google Hangout good?
- All 5 scientists said yes
Did you use the training during the event?
- 3 said no, because it didn’t come up, and they didn’t want to mention it unless they had to
- 1 said yes, finding themselves using the training many times during the event
- 1 said a little, but they didn’t get many questions on it
Did it make you more confident to talk about animal research?
- All 5 scientists said yes
- 1 of the scientists mentioned they were wary of talking about animal research in the live chats, in case time ran out and they couldn’t explain their work properly
Would you recommend it to others?
- All 5 scientists said yes
How would you improve it?
- Open the session up and ask for individuals input more
- Links to resources that show that animal experimentation is not ‘animal cruelty’
- Have more scientists who use animals in their work, to encourage more open discussion among peers
And a few other comments:
“Very helpful and answered a lot of the questions I had about discussing animal research with the public”
“I have contacted UAR and organised a school visit of my own!”
Wellcome Trust engaging science day
How to share and disseminate the learning from your project and evaluation.
On 14th May the Wellcome Trust held one of their twice-yearly Engaging Science days. They’re an opportunity for grantholders to meet with each other and Wellcome staff, to share learning and ideas. And to sample Wellcome’s renowned lunch offerings (side salad complete with edible flowers and quail eggs, anyone?).
Gallomanor were invited to speak on the panel of the session convened by Ben Johnson of Graphic Science, with the blurb “Once you have completed your project how do you widen its impact by ensuring others learn from what did (and didn’t) go right? An effective evaluation can also help with securing further funding and a wider roll-out of your project.” Other panelists were Manisha Lalloo from the Royal Academy of Engineering giving a funder’s perspective, and Becky Parker & Dave Colhurst from Simon Langton Grammar School.
Evaluating the learning from our projects and disseminating it (both within the office to colleagues and to the wider SciComm community) is important to us, so this is a summary of what I spoke about in the 5-10 min slot:
Angela sums up my talk excellently
- Be brave in budgeting. If you think an external evaluator will help, budget for it. Plan honestly for the amount of staff time your evaluation process will take.
- Write up and talk about your findings, however small they seem. We all have to report back to funders, but making these reports readable will encourage others to read them. Start with a concise executive summary, use visual cues to break up text. Beyond this, share what you’ve learnt with colleagues and peers. We have an evaluation site where we post snippets of learning. Think about who you’re writing these for and target them – funders, participants (scientists, teachers, general public), or practitioners?
- Plan viable dissemination routes. Be innovative with evaluation and people will notice and ask you to speak and share learning at conferences. We’ve found this a brilliant way of publicising our projects, and it’s bought in new funding from people in the audience, such as the online discussion zone for the Ri Life Fantastic CHRISTMAS LECTURES.
And my top tip for dissemination when Ben sprung the question on us? Be short and visual in reports.
I’m a Scientist zones for primary students
We’ve had plenty of primary schools take part in I’m a Scientist zones in the past, nestled in among secondary schools. Feedback from primary teachers and students shows they get a lot out of taking part. Primary schools are looking for science enrichment activities too, and talking with real scientists is exciting at primary school as well as secondary. Scientists often wouldn’t know the students were Year 5 & 6, not Year 7 & 8, if we didn’t tell them.
This March we decided to run 2 primary school only zones, for Year 5 & 6 students. In part to avoid possible situations of primary students reading questions on non-primary-friendly topics asked by secondary students in their zone (think sex, drugs, rock’n’roll). Partly to open up I’m a Scientist to a wider group of UK schools and students. And importantly because one of the conclusions from the ASPIRES project about young people’s science and career aspirations, is that STEM education projects need to begin earlier, at primary school.
The 2 primary zones – Caesium Zone & Xenon Zone – were general zones, each with 5 scientists from a range of research areas, but avoiding any non-primary-appropriate topics (no IVF experts for instance).
What we learnt:
- There is demand for primary zones. We had too many teachers wanting places and had to turn some away. We’ll be running more primary zones in future events.
- Primary zones aren’t that much different to secondary zones. They felt like just any other zone. We don’t need to change the format for primary students. The main difference we saw was that the questions were often more factual than conceptual, live chats were less disruptive than with older students, and students left lots of comments thanking scientists for their answers.
- Being in a primary zone didn’t seem to affect the scientists’ experience of taking part. When we offered them a place we mentioned it was in a primary zone but that shouldn’t change how they approach engaging with the students.
- Online safety is more of a worry for primary teachers and parents. We need to be clearer about how secure the site is and giving advice, such as students not using their first and last names in their username. This is true for secondary zones, too.
“Did you always like science when you were in primary school”
How does I’m a Scientist affect students’ attitudes to science?
We know anecdotally that participating in the I’m a Scientist event has a positive effect on students’ attitudes to science. Teachers tell us that their students were buzzing, and that they understood more about science. Students thank scientists in live chats for an interesting lesson. They tell us that it is “better than Facebook”.
But being data geeks that isn’t good enough. We wanted to know just how much we were affecting the students. Were we affecting them all by the same amount? Girls, boys, year groups? Did being more active in the event mean a greater change in attitude?
Questions we ask students
Thanks in part to the pilot work by Robin Longdin, then a SciComm masters student at UWE, we ask every student registering for I’m a Scientist a set of 4 questions:
- How does school make you feel about science?
- Are you planning to choose a science subject at the next stage of your education?
- Do you think jobs involving science are interesting?
- When you finish your education, how likely are you to look for a job that uses your science knowledge and skills?
We also ask the same questions towards the end of the event and ask for the username so we can match against their initial answers, their gender, year group, zone and activity levels within the event.
Measuring attitudinal change
The important thing for us is not the answers they give but the difference between the beginning and end of the event. We want to know if after two weeks of being exposed to our scientists they feel differently.
Each question had a five point answer scale. We intentionally wanted a neutral middle answer. We can’t expect students to all feel passionately about science. We then allocated a numerical value to each answer. 2 for the most positive, -2 for the most negative. We then simply subtract the starting answer value from the end of event value to give us a value for change in attitude for each of the four questions for each of the student who answered.
Do more active students see a greater change in attitude?
There is a lot of data generated and one of the challenges is to consolidate it into meaningful clusters. For example what do we mean by levels of activity? Some students are lively in live chats. Others ask a lot of questions. Some do both. We looked at the 3 main types of activity: ASK, CHAT, VOTE and scored each student between 0 and 3 depending on how active they were. We then totalled the individual scores to give each student a score between 0 and 9 overall. That is the X-axis below.
The first thing to note is that the overall trend is that the more activity on the site the more positive the change in attitude. Secondly the change in attitude to the final question: “When you finish your education, how likely are you to look for a job that uses your science knowledge and skills?” is much lower than the other questions, and in fact sometimes negative even for those students who made the most of our erstwhile role models.
We need to investigate this further. It could be a flawed question? What does it really mean to 13 and 14 year olds. Do school students consider a PhD studentship to be a job? Are half our role models not helping with that question?
UPDATE:
We went back to the source data and found to our embarrassment that we had made a mistake and we were comparing the answers to the final question to the answers to the 3rd question. D’Oh! The good news for us is that when you use the right data, the story looks even better. Students emerge from I’m a Scientist feeling much more positive about jobs in science for themselves.
The greatest effect is on girls and Year 10 & 11 students
This graph shows the average change in attitude for 6 groups by question asked. It clearly shows that the event has the greatest effect on girls and on students in Years 10 & 11.
This is based on a total of 853 valid responses from students across 3 events from June 2013 to March 2014 in the UK. The n for the groups in the second chart are:
All: n=853
Girls: n=475
Boys: n=333
Primary: n=90
Years 7-9: n=589
Year 10 & 11: n=130
They don’t all add up to 853. Sometimes it is a group of girls and boys who register. We also had 21 6th formers respond, but that is too small a sample to use. And sometimes the default “Please choose” answer remained. Yes there are disproprtionately more girls than boys, but yes, more girls than boys do participate.
Leave a comment and start a conversation!
Finally, we do this analysis for our own sake so that we can better understand our event, but we publish it for your sake, whoever you are. We want other practitioners to share in our knowledge. Please leave a comment to say “thanks”, or to ask for clarification or for more information.
March, June or November?
When teachers ask me what aged students they should bring on I’m a Scientist, my answer is often something along the lines of “well, it’s developed for Year 9 students but all ages get something different out of taking part, so take your pick”.
The good news is that our data supports this. When students register we ask them what year group they’re in. There are Year 5s. There are also Year 13s. Around two thirds of the 30,000+ students registered so far are KS3 (Years 7, 8 & 9).
The bad news? It’s not really bad news as such, but it looks like I should also be advising teachers on the time of year they should take part with different year groups.
March and June are great for pre-GCSE students. Teachers tell us they’re looking for something fun to do with students at the end of the year. They’re less good for GCSE and A Level students who have mocks, coursework and shock horror, actual exams to contend with.
November is slap bang in the middle of careers time for older students, and who better to speak to about possible careers in science than 5 practicing scientists online?
Different year groups take part at different times in the year
And just in case you’re wondering why there are a suspiciously high % of Year 5’s – that’s the registration form’s default option. So you might even expect it to be higher!
Girls vs Boys
I was asked recently what the gender split was for students taking part in I’m a Scientist. I replied that we’d never looked on the assumption that since teachers took their students online it would reflect the school population.
I’m never one to turn down the chance to crunch some numbers so I took a look. And was surprised. At registration we ask if the person registering is a Boy, Girl or A group of students (where more than one pupil is sharing a log-in). We have data going back to June 2012 and nearly 15,000 respondents to the question.
Surprisingly it wasn’t a straight 50:50 split. So I then decided to look at it by year group.
What we saw was that year 9 & 10 were significantly skewed towards girls. And that bias continued through to 6th form.
Is that common for science outreach events?
I’m a Scientist so far in numbers
How many zones have we run? How many students engaging with how many scientists? Asking how many questions? What year group are they? Where in the UK are they?
The first slide of the I’m a Scientist summary so far…
These are some of the questions we get asked a lot, and until now haven’t had all the answers in one place to roll off the tip of our tongues. We’re pulling together some slides summarising I’m a Scientist to date and once they’re done we’ll post them up here.
In the mean time, I’ll be posting snippets that show just how far we’ve come since our first big Wellcome Trust grant in 2010.
To start us off (and to set the scene for the graphs and maps to come in future posts) here are some of the numbers so far, up to February 2014:
Zones: 118
- 51 general science zones, 67 themed zones
- Spread across 10 events, in March, June & November each year
Scientists: 590
Students: 36,225
- % of students that actively took part: 84%
- Average number of students per zone: 307
- From 462 schools
Number of questions asked: 86,645
- Number of questions approved: 38,836
- Number of answers given: 84,890
Live chats: 1,616
- Lines of live chat: 528,035
Visits to imascientist.org.uk: 1,592,016
- Unique visitors: 1,306,978
- Page views: 4,958,725
- Average visit duration: 3:02 minutes
Visits to IAS by technology type
As part of a review of what we need to do over the next 5 years with I’m a Scientist, we looked at how visitors accessed the site and how that has changed year by year over the last 4 years. We also looked at how visitors registered as students differed from All visitors. The Student visitors probably give a fair reflection of school IT capabilities. In the graphs the student data is represented by dashed lines. We only have student data from 2011 when we started tracking registered users separately.
It was no surprise to see that visits from mobile and tablet devices were increasing but having 23.6% of visits from mobile devices in 2013 was a shock. Delving deeper it is clear that it is mostly questions that mobile users visit. Tablets tell a different story. For student tablet users there is little difference in content visited when compared to student desktop users. And it is interesting to see that the levels of tablet use are the same for students as for all users at just under 8%.
We need to adapt our design for the question page so mobile users get a better experience.
We also need to be more aware of how easy it is for registered tablet user to interact with the site.
Browser usage tells an interesting story. Unsurprisingly Internet Explorer is plunging. But at over 60% it is still the most popular browser in schools. More surprising was the drop in usage of Firefox. The rise in other browser is mostly due to mobile and tablet browsers with Android, Blackberry and Opera featuring.
Finally the web developers enemy – old versions of IE. IE6 has just about disappeared. Yay. IE7 hangs on at 4%, but IE8 remains popular. The surprise for us in this graph was that All visits and Student Visits differed little in respect of IE version.
If you have any questions or would like more data. Please let us know in the comments or via email.
I’m a Scientist acts as a public engagement booster for scientists
Anecdotally, we’ve heard of how I’m a Scientist can be a good starting point for science communication activities. This was the case of Suzi Gage, Tom Crick or Suze Kundu, who took part in I’m a Scientist in June 2011 and are now putting a great emphasis on the communication side of their scientific careers or even fully devoting to it.
Twitter conversation about the impact of I’m a Scientist on Science Communication careers
However, we were still curious to know to what extent we could extrapolate this to the wider community of scientists that have participated in the event during the last years. How could we know if I’m a Scientist had encouraged them to do more science outreach? Well, we decided to ask them.
We sent a survey to all the scientists that had participated in I’m a Scientist until 2012, leaving a gap of at least one year since they took part in the event. Approximately a quarter of the scientist responded to the survey, resulting in a more than decent sample of 113 scientists. The data collected in the survey show that there is a strong indication that I’m a Scientist is a real public engagement (PE) boost for scientists.
Summary of the survey’s main findings
I’m a Scientist is a good way to start doing public engagement in schools
If we look at public engagement at schools, we have recorded a significant increase, especially among those scientists that were particularly new to this form of outreach. Moreover, scientists who had never done school public engagement were encouraged to do more public engagement in general, going from none to an average of over 4 activities per year. This was supported by some great comments gathered in the survey:
Having never done outreach with schools before, IAS gave me the chance to engage with a different audience than I would typically.” – Scientist
Scientists find the event flexible, open and inclusive
The online nature of the event was praised by different scientists that left comments in the survey:
I really liked how the online format broke down barriers and allowed the students to ask anything they wanted without having to stand up in a crowd.” – Scientist
Having all the activity online also gave me the flexibility to contribute more of my time, ad from a remote setting, compared to face-to-face school visits.” – Scientist
“Sifting the evidence” Suzy Gage blog hosted by The Guardian
Some of the scientists had already contributed to the online scientific community with blog posts, podcasts or through social media. However, for a high proportion of them (68 out of the total 113) I’m a Scientist represented their first online public engagement event, which really pushed up their participation in general public engagement. In this case, their collaboration with public engagement activities went from zero to more than 3 general outreach events per year.
It was also interesting to find out that those who had already done lots of public engagement (4-15 activities per year) started doing more specific online outreach after participating in I’m a Scientist, at the expense of other forms of public engagement.
I’m a Scientist reveals itself as a great launchpad for budding science communicators
The majority of the scientists that filled in the survey (86%) had already taken part in some kind of public engagement activity (lectures, science festivals, interviews in traditional media, science policy making, etc.). It was a nice surprise to find out that scientists who had done very little public engagement (1-3 activities per year) increased their activity dramatically, going up to 5-8 outreach activities per year after the event. What is more, the greatest increase in overall public engagement activity (an increase of 130%) corresponded to the scientists who were just doing very little of it before the event.
Looking at the big picture, there is a general trend that I’m a Scientist enhances the participation of scientists in outreach events, except in the case of those who already did loads (more than 30 activities per year), where there is very little room for improvement. All in all, we are happy to confirm that I’m a Scientist represents a great launching platform for public engagement in science.
It remains the best public engagement event in which I have been involved.” – Scientist
modangela in Evaluation, Event News, News, Science Engagement, Scientists |
Leave a comment
How does I’m a Scientist change students’ perceptions of science?
We’ve just run I’m a Scientist in Ireland and are curious to know how our event actually affects students’ attitudes towards science.
In order to do this, we included a short and compulsory pre-event survey in the form students used to register. We then asked students to fill in the exact same survey on their profile page after the event. When we matched the data from the two surveys, 92 students (7% of 1,247 students that participated in I’m a Scientist) had filled in both surveys. Importantly, data from the total number of students that filled in the pre-event survey very closely correlates with the pre-event data of this 92 student sample.
We were very happy to find out that students’ interest in science and science related careers is clearly increased after taking part in I’m a Scientist. This is what we have learnt:
The amount of students that say they love science doubled after taking part in I’m a Scientist
How does school make you feel about science?
I’m a Scientist really got the students excited about science! Before taking part in the event, only 23% of students said they loved science, but this number increased up to 51% after the event. On the other hand, the number of students that don’t feel really excited about science or think it is boring decreased from 9% to 3%.
Participating in I’m a Scientist encourages students to choose a science subject in the next stage of their education
In the pre-event survey, 66% of the students were absolutely certain or very inclined to choose a science subject next year. However, this percentage raised up to 71% after participating in I’m a Scientist.
Thanks guys for talking to me really helped me make my decision for the leaving cert – sarahlawless, student
Students are keener on science related jobs after participating in I’m a Scientist
The majority of students that completed the surveys already thought that jobs involving science are at least fairly interesting before taking part in the event, but there was still room for improvement and the percentage of students that considered science related jobs very interesting saw a big increase from 36% to 62%.
Taking part in I’m a Scientist increases the likelihood of students looking for a job that uses their science skills
How likely are you to look for a job that uses your science knowledge?
Before taking part in I’m a Scientist, a big portion of the students (37%) couldn’t decide whether they would try to look for a job that uses their science skills and only 10% said they were sure that they would look for this type of job. However, the event seemed to be the boost that students needed to be more confident about looking for a science related job. After I’m a Scientist, most of the students (68%) said that they would certainly or very probably look for a job that uses their science knowledge.
Honoured to have taken part. The future of science is in very good hands with you guys! – scientist
In addition to this, students left several comments that stated clearly how they were enjoying and learning at the same time throughout the event. They liked that the event was so interactive and that they had an active part at every step: asking, commenting, chatting and voting.
In the future, we would like to use this same strategy to measure the impact of other events or activities. We are also very interested in analysing the gender and year course differences that could be found in the pre-event data.
modangela in Evaluation, Event News, IAS Event, News, Science Education, Science Engagement, Teachers |
Leave a comment
IAS Live: Antibiotic Awareness Day
Yesterday afternoon, on European Antibiotic Awareness Day, we joined the Royal Society of Chemistry (RSC) at Burlington House in London, Piccadilly for a foray into antimicrobial research.
I’m a Scientist: Live at Burlington House.
We took five scientists — three of whom were I’m a Scientist alumni, having taken part in previous online events — whose research looks at the uses of antibiotics and antimicrobials, put them in a room in-front of around 100 sixth-formers, and had them answer questions on everything from Typhoid Mary, to zombie apocalypses.
Our scientists were Rob Shorten, Jess Bean, Mark Roberts, Clare Taylor, and Emma Newton.
Jess Bean and Mark Roberts go for the BAFTA puppeteering award with plush microbes.
Helen Arney, our MC for the afternoon kicked off by introducing the scientists before launching into a round of Science Fact or Fiction. The facts covered everything from bacteria flavouring cheese and yogurt, to our being only 10% human, with most of the cells in the body (by number) actually being bacteria. Clare introduced the notion that around 3% of the population can carry Salmonella their whole lives without ever becoming ill; setting up a whole round of questions on historical poisoning and Typhoid Mary.
We then moved on to the questions round: Emma presented a world first, showing us her “electronic nose”, a prototype device for sniffing out different bacteria based on the chemicals they produce. Using plush microbes, Jess and Mark acted out an Academy Award worthy demonstration of a bacteriophage’s attack on an E coli cell. Rob took us through the purposes of a faecal transplant, Mark explained how some bacteria can follow the earth’s magnetic field, while Clare took the audience award to the best piece of advice for surviving a microbial zombie outbreak: “there’s always a percentage of the population with a resistance; seek them out and make a cure.”
In the end there could only be one winner. After giving the scientists a last plea for votes the votes were counted yielding a surprisingly close spread, with Clare standing above the others to take hold the coveted I’m a Scientist: Live trophy and mug.
Let’s evaluate
Standard end of event competitors’ photo, accompanied by MC, Helen Arney.
The room was filled, with around 100 students in their seats, votes being counted using the electronic voting system were all around the 90 mark. The audience was made up predominantly of school students, with about 12% being teachers.
The use of electronic voting pads gave us the chance to do some on-the-spot evaluation of how the audience found the event. A majority of the audience who provided this feedback said they had learnt something with 49% saying they had “learnt a lot”, and 37% learning “a bit”; only 8 respondents (14%) said they had not learnt anything.
To the question, “Has the show made you want to find out more about antimicrobials?”, 50% of those who replied “got the bug”, while 18% had had enough. Owing to a slight technical fault we weren’t able to count the replies to, “Did you enjoy the show?”, what we did get were a whole lot of positive comments from students and teachers.
Really informative; loved the bit when the scientists talked about their specialties; really interactive – the pads were great; great venue and they liked the refreshments (felt very adult!)
Many students stayed behind to ask more questions to the scientists once the event was over, and working by the buzz in the room, and the comments from teachers and students, it seems like the afternoon was a resounding success.
What have we learnt?
- Scientists bringing props is a good thing.
- Being nice and altruistic can get you lots of votes, or certainly lots of applause.
- It’s useful to run through the technical aspects of the event maybe a couple more times than might feel necessary; can’t be too prepared.
- Students appreciate when a refreshment buffet includes cake.
Moderator - Josh in Evaluation, I'm a Scientist - Live |
Leave a comment
IAS Live: Drugs, Bugs and Infections
Last night five scientists researching different aspects of antimicrobials took to the floor at the FIS Conference at the ICC in Birmingham to take part in I’m a Scientist Live: Drugs, Bugs and Infections.
Photo courtesy of @biologyatkinge1
Our five were:
Photo courtesy of @PharmaMix
30 mins later with time running out, Simon brought the event to a climax by asking the scientists to spend 30seconds telling the audience why they should get the votes and the winner’s trophy. The vote that followed confirmed that Anne Marie’s promotion of the IMI’s Summer School for Year 12 students was enough to edge her into the lead and win the vote.
Some numbers:
181 tickets ordered.
105 attendees on the evening (105 voting handsets were used).
Over half the audience were students evenly split between schools and undergraduates. This means a lot of the students who asked for tickets failed to show. We had a capacity of 150 for the evening with a maximum limit of 200 for the room. Whilst we expected some dropout we are surprised by the level of no-shows. We will need to over-book more heavily in future.
Voting was close. Only one vote separated Anne-Marie from the scientist who was second-placed. We discussed what to do in case of a tie beforehand, but were clear it wouldn’t end up as a draw. We need to take those plans more seriously in future.
Having the electronic voting cards allows us to ask some immediate evaluation questions. A healthy majority enjoyed the show and learnt from our panellists. A slightly smaller majority have caught the antimicrobial bug.
What else did we learn/have confirmed from last night?
- Allowing a 30 minute turnaround between a lecture and IASLive with only one sound engineer is a little optimistic.
- We should brief participants that they will be using a lapel mic and that they therefore need a lapel.
- Some sort of stage helps if the audience are on the same level
- Enthusiasm and energy will give rise to a decent discussion whatever the subject.
What criteria do students judge scientists on?
When students take part in I’m a Scientist they get to vote for their favourite scientist to win £500 to spend on communicating more science. It gives them ownership of the project and they decide who gets, in effect, a small grant for public engagement. But what are students’ votes based on?
This is how 855 students (or groups of students) have ranked certain criteria from most to least important when considering how to vote for scientists, since January 2012. The results come from a Drag & Drop ranking activity in the first lesson plan “You’re the Judges” that teachers run to introduce their students to I’m a Scientist.
The top ranked criteria are all linked to health or the scientific method. This is reflected in the number of questions students ask scientists about saving lives and animal testing. Reassuringly, the more superficial criteria lurk at the bottom of the list.
Other criteria that students suggested include:
“Abides to Scientists ethics, and morals”
“A person that thinks ‘outside the box’. Confident person. Optimistic person”
“Enthusiasm”
“How much they believe in and care about their work”
“The speed that they answer my questions. Also if they turn up for a chat”
“How great their beards are”
“If the scientist works at weekends (as well as during the week)”
And finally, the crux of all scientific research:
“Whether their tests are fair or not”
One teacher commented that, “the class disagreed on the ‘good-looking’ criteria because some feel that people are judged by first impressions and looks are included in that”.
Teachers, how do these criteria compare to how your students judge the scientists? Did students generally agree on the rankings or did certain criteria provoke more discussions than others?
Using teacher feedback to plan future themed zones
A question I sometimes get asked when I tell people about I’m a Scientist, is how do we choose which zones to run? In short: we ask teachers what they’d like.
In July we emailed all the teachers signed up for I’m a Scientist to ask them what zones they’d like to see in our next three events: November 2013, March 2014 and June 2014.
Just over 40 teachers filled in each survey by ticking as many of the 40 zone choices as they’d like (some are zones we’ve run before, others are new).
Which zones came out top?
The graph shows the top 20 zones, by the number of teacher requests across the 3 surveys for November, March and June. Forensic Science and Genes zones came out top, closely followed by Stem Cells, Health, Organs and Sport Science. Most of the top requested zones are biomedical; this might be because there are more Biology teachers on our list, due to the majority of funding coming from the Wellcome Trust since 2008.
When to run these zones?
What’s more interesting is the variation when teachers want to see certain zones, reflecting when topics are taught in the school year.
- Stem Cells, Drug Development, IVF and Subatomic zones were more requested in November than March or June.
- Earth, Chemicals and Technology zones were less requested in March than November and June.
- Communication and Human Limits zones were more popular in June than November or March, while Blood and Reproduction zones were less popular.
So what…?
We’ve used this feedback to choose some of the zones we’ll run over the next year in advance. This will help us plan teacher and scientist recruitment throughout the year. STEM contract holders have also told us it’s useful to know dates and zone themes in advance.
And sad news for any linguists out there, I don’t think we’ll be running a Language Zone any time soon. With only four requests there just isn’t the demand.
A taste of the themed zones running over the next year is at imascientist.org.uk/2013/10/im-a-scientist-zones-november-2013-june-2014. We’ll add more themed zones from other funders to the mix nearer to the events.
We’ve published all our zone reports from June
We’ve published all our individual zone reports from the June 2013 event. If you delve further back into the archives, you’ll find them for the zones we ran in March 2013 & November 2012 too. All the reports are tagged (funnily enough) under “Evaluation Reports“.
Each report sums up the activity in that zone. They provide some information for the scientists and teachers in that zone about how much the students and scientists interacted.
How many questions were asked and answered? What were the popular topics asked by students? What examples of great engagement stood out? What did students, scientists & teachers say about taking part? How does the winner plan to spend their £500 on communicating more science?
We email out each report to all the scientists and teachers in that zone. Scientists tell us the reports really help justify their participation to supervisors and colleagues. The detail tells them far more than just receiving a participation certificate in the post. The ‘loads’ of questions they answered turns out to be in the hundreds. The live chats went in a flash thanks to around 300 students bombarding them with questions over the 2 weeks. And the students really did ask about their research – volcanoes, bacteria & cancer for the zone below!
Take a look. If you’ve got any comments, such as what other info might be useful for scientists and teachers, we’d love to hear them.
Who are you Twitter?
As we prep for our Science Communication Conference session on how we evaluate I’m a Scientist, the occasional sidetrack leads to interesting stuff.
A while back we used Tony Hirst‘s scripts and know how to download our twitter followers network. We’ve used Gephi to map out the clusters within our network which helps us work out who is following us and how diverse our network is. We were delighted to see an Australia cluster thanks to our friends in Adelaide. More about that later.
Another useful benefit of downloading your followers is the ability to search their bios to find specific followers who are interested in certain subjects. This evening we identified 83 followers with “food” in their bio. We’ll be contacting some of them (excluding those who are just “foodies”) to tell them about our new debate kit on Food Security.
Whilst I was there I thought I’d create a little wordle with big word in it!
100 word Wordle of Bio of followers of @imascientist
To create this wordle I exported the Gephi table into Excel and converted the bios into lowercase and change sciences into science. More cleaning of similar words would create a better wordle but time is always short. Then I pasted the words into Wordle, set a maximum of 100 words, and removed some of the irrelevant words such as twitter and tweet and like. Set some Custom Colors and tweak the layout until it worked and you can see the result. Enjoy.
IAS Outreach by STEMNet Area
We’ve selected the scientists and schools (all will be revealed next week) and we’ve looked at where they are across the country. We want to work closely with STEMNet contract holders to promote our events to teachers, scientists and engineers, so we’ve calculated how many schools, classes and scientists are in each STEMNet contract area. From that we’ve calculated the estimated number of interactions (or Live Chats as we call them) we expect per area. On average each class has one live chat and each scientist attends seven.
The best news was that there are only 3 areas where there is no activity. We are doing science engagement and enrichment across the country.
But where do we have most activity?
Most classes signed up:
- Bristol, Bath and Somerset – 25
- Lancashire – 19
- Surrey, Central and East Berkshire & West Yorkshire – 18
Special Kudos to Liz Lister in Bristol, Bath and Somerset – over 13% of secondary schools in the area are signed up.
Most scientists:
- London North – 6
- Bristol, Bath and Somerset – 4
- Greater Manchester – 4
- Oxfordshire, Buckinghamshire and West Berkshire – 4
- Surrey, Central and East Berkshire – 4
We are very keen to work with all STEMNet Contract Holders to get their STEM Ambassadors taking part in the event and getting schools in their area enjoying the benefits of reaching scientists from across the country and world.
We have a promotional pack that we can send out containing a flyer, video and sets of postcards for both teachers and scientists.
IAS Postcards from the Promotional Pack
Please contact us if you would like help in getting more STEM Ambassadors and schools involved – rosie@gallomanor.com or 01225 326892.
What themed zones do teachers want?
For I’m a Scientist to work we need to run zones that teachers and students want. So when we’re deciding on zones to run in the next event we ask teachers to tell us what they want.
Teachers voted on a longlist of zones we drew up – they could select as many as they’d be interested in taking part in. We also asked them for suggestions for other zones.
Over 100 teachers told us what zones they’d like to see. The results are below.
IDACI Index Profile
It it important to know who we are reaching with our events. Funders are keen to know and we want to know that we aren’t just reaching the usual suspects. We’ve recently done some geographic analysis, which we’ll share soon.
But general location is only part of the story.
Nice schools in nice areas tend to do plenty of science engagement and send lots of students to university. Therefore it is important to see if we are reaching those schools who don’t normally participate; those hard to reach schools.
But how do you measure that?
Does the deprivation of the area in which the school is located provide a good proxy? Some studies would suggest so: “The Characteristics of Low Attaining Pupils” DfES Statistics Bulleting 02/2005 June 2005
We plotted the IDACI Score of the schools who participated in IAS and compared the profile (based on deciles) against that for all schools. You can match postcodes to IDACI thanks to a nifty tool from the Department for Education
The problem is that most schools are in an area with a score of between 0.1 and 0.3. It isn’t granular enough to see much difference. BTW the lower the score the less deprived the area. I suspect this is because most areas also fall into that range.
So we then looked at the ranking. IDACI area is ranked between 1 (most deprived) and c. 33,000 (least deprived). We then grouped into groups of 3,000 and plotted a histogram. This time the All Schools distribution is much more even allowing us to spot differences between IAS and the national distribution.
And so what?
In the most deprived quarter of areas we follow the national profile, but midway we slip behind and find ourselves with more schools than expected in the least deprived areas. It is no surprise. It is our expectation that schools in the least deprived areas will be more likely to find time to seek out and run more engagement and enrichment projects.
But,
I really want to know how our profile compares to other science engagement projects. Are we all hitting the same schools? Are there differences that we can build upon to ensure the widest reach?
Please use the comments to give us any feedback or suggestions of where to go with this.
SpotOn London 2012 – Background
As part of ScienceOnline (or SpotOnLondon2012) Karen Bultitude and I have arranged a session called: Can we work together to better evaluate online engagement? It’s taking place in the Steel Room on Monday 12th November at 2.30pm.
We wanted to provide some context to the session. Here goes:
Since you’re interested in ScienceOnline it’s fairly likely that you think online science engagement is important and going to get more important. Me too. I run a few online science engagement projects, https://imascientist.org.uk is the most prominent. It’s pretty important to me to know how well the project is performing. We need to know in order to improve it, we need to be able to tell our funders and we feel obliged to be able to demonstrate to all stakeholders that the project is effective.
So we evaluate our projects using a wide range of methods: surveys, interviews, personal meaning mapping and we also try to analyse the vast quantities of data that we capture as a matter of course from the website and Google Analytics. We use that information to work out how we’re doing.
All well and good except that without anything to compare against it is difficult to know how we are doing. A starting point when wanting to compare against sector standards is knowing what to measure. It is surprisingly difficult to find much information about how online engagement properties are performing. We try to publish all our evaluation. Google: evaluation I’m a Scientist. The first three results are our main evaluation reports and more informal evaluations of two spin-offs. Try it for another online engagement project. Google: evaluation [INSERT PROJECT NAME] – very little seems to be published.
The Cultural Sector (including The Science Musuem) recognised this problem in 2009. They came together to produce the “Let’s Get Real: How to evaluate Online Success” report in order to better understand their digital activities and to help them plan future activities more effectively in more financially constrained times.
The report brought together 17 museums, galleries and venues. They shared statistics and methodologies. They agreed a common configuration for Google Analytics for the purpose of reporting to funders. They shared good practice.
There is plenty of interest in how we measure our online activity. This session is the 3rd on the subject in this conference alone. Our hope is that we can work together to better measure our online success.
There will be benefits to better measurement. We’ll be able to improve what we do. We can more more efficient, more effective. We’ll spend less time wondering what, out of the mountains of data, is important. We’ll also be able to demonstrate to funders in a coherent and consistent manner that online engagement is effective. And that is something that will benefit all of us who believe in online engagement.
We've cut our teacher survey from 41 to 18 questions
This morning I cut down the survey we ask teachers to fill out after taking part in I’m a Scientist. The old survey was a hefty 41 questions long, and the new one has just 18 questions.
By looking at teachers’ previous responses to the 41 questions we were able to identify the questions which worked, and the questions which didn’t. Using this information we’ve made the survey simpler.
It’s also more quantitative. Funding from the Wellcome Trust for the next 3 years allows us to improve the way we evaluate I’m a Scientist, and move to evaluating outcomes through more quantitative measurements.
My next task is to do the same for the scientist and student surveys, and cut them down from 28 and 25 questions.
Evaluating the Impacts of engagement
How can we evaluate the impact on students taking part in I’m a Scientist? Can we measure if they’re more likely to take a STEM subject at A Level? If they’re more likely to study science at University? How should we use the large amounts of data generated by online projects? How can we share our evaluation in a more useful way?
These are just some of the questions we’re trying to answer about evaluating I’m a Scientist and other Gallomanor run projects. Judging from the first in a series of seminars looking at Evaluating Impacts of Public Engagement and Non-Formal Learning, last Friday 4th November, others are thinking along the same lines.
The Core Issues & Debates seminar kicked off the series at the Dana Centre in London, and bought together a range of researchers, evaluators and learning and communication practitioners. Future seminars focus on areas such as how to reach new audiences, evaluating online engagement and using qualitative evaluation methods.
The 7 speakers approached evaluating impacts from different views – funding, strategy, science festivals, academic, and museums/science centres. There were some key themes that emerged during each of the 20 minute talks and the resultant Q+A sessions. (It would have been useful to have a bit more time for Q+A discussion after each speaker, as the allocated 10 minutes were quickly eaten into.)
- Evaluation needs to be shared with others so all projects are ‘learning projects’. The British Science Association’s Collective Memory is a good place to start. It’s worth constantly thinking about how to improve evaluation during a project, such as changing evaluation questions so they return more useful responses.
- Evaluation is very important right from the grant application stage at the start of a project, but shouldn’t be done for the sake of it, or just because funders ask for it.
- There are still lots of questions unanswered about how to evaluate and measure the impacts of an engagement project. Is it really possible to measure if students are more engaged with or interested about science as a direct result of one activity? Is it enough to accept your activity is one of many factors that may have influenced a change seen? These will hopefully be explored further, and maybe even answered, in future seminars in the series.
- Negativity can be hard to capture in evaluation. Evaluation studies can therefore be designed to try and capture negativity, such as framing questions to encourage participants to think not just about the positives of the event.
- Bad evaluation that draws inaccurate or invalid conclusions from data can be more damaging than no evaluation.
Overall it was a useful introduction and summary of how impacts are being evaluated. Armed with my 7 pages of dense notes scribbled during the seminar we’re now working out how to put some of these ideas into practice with I’m a Scientist. This will likely spark another post in due course.
Click on the RSS symbol at the top left of the page to subscribe to the blog, or register for email updates.
Read about our session at Science Online conference
We’d never been to the Science Online conference. And we’d never given a presentation about I’m a Scientist, Get me out of Here! But when the organisers asked if we’d like to do a session on the event we jumped at the chance.
We then closed the office for the whole of August, so we had to write and organise our highly interactive, possibly risky session in 2 days when we got back last Wednesday. At the same time as do all our catching up from the holidays.
Last week was an interesting week…
So here is a summary of our session (last Fri, 3rd Sept) and what we, and others, said.
Our presentation
Sophia presenting at Science Online coference London
At Gallomanor we like to walk the walk, as well as talk the talk. So we wanted to make the session as interactive and audience-led as possible. After a short (2 min) intro, we showed the audience the 12 topics we’d prepared and asked which 6 they wanted to here about.
Here’s the prezi presentation, with a slide on each topic. I’ll summarize below what we were going to say on each – including the ones we didn’t cover on the day.
1. Film – we showed our ‘Intro for the classroom’ film.
2. Our Philosophy
The audience didn’t vote to hear about this, which was a shame as I think it’s really important in explaining why we do what we do, and how. Fools, fools!;-) Maybe we should have called it something more exciting sounding…
Anyway, key points would have been:-
- We reverse normal power structures (kids in our event are given some decision-making power and the chance to ask what THEY want to ask). This engages pupils who are normally turned off. See more on my thinking here in this piece on the Secrets of Engaging Teens.
- Making it fun and game-like is not a gimmick and doesn’t make something not educational. It makes people pay attention and be interested! (e.g. this research)
- It’s not just about getting kids to study science. Not all teenagers will grow up to be scientists, but they will all grow up to be people, and need to have a relationship with science. More on this point in this article for Wellcome Trust blog.
- Scientists have a lot to gain from engagement too – they aren’t just doing everyone a favour. They can be energised, challenged and made to think by the huge variety, and inventiveness, of students’ questions.
3. History of event
Shane thought of the idea for our sister event I’m a Councillor, Get me out of Here! while drinking Guinness. It’s been running since 2002, helping councils and local councillors to engage with young people in their area.
We came up with the idea of doing a science version in 2007 and ran the first, pilot event in 2008.
4. Funding
The pilot was funded with a People Award from the Wellcome Trust. We then successfully applied for a follow up grant from Wellcome’s Society Award scheme.
We love Wellcome and they are fab. We approached lots of people – people like BIS and Dept of Education who you’d think would be the right people to approach, and we got nowhere. All roads led back to Wellcome.
If you’d like to apply for science engagement money from Wellcome, our advice would be:
- Make sure you understand what they value and what they are looking for, and that your project fits in with that.
- Contact them early and talk to them.
- Make sure you have really thought it through and explained your plans (not just vaguely). They will fund imaginative, even risky stuff, but they want to know you aren’t going to piss the money up the wall.
- Realise the importance of formative (and summative) evaluation.
5. Site detail
Shane was just going to run through what’s on the site and how it works. You can prob just have a look for yourself.
6. Site build
It was in WordPress, cos we like and support open source, etc. And also cos WordPress is great. Apparently the site is doing things with WordPress that no one has done before. This is of course very exciting.
Developed by total legend Mike Little. Here’s Mike’s presentation about the project at Wordcamp 2010.
7. Scientist Recruitment
We pursued as many routes as possible to recruit scientists, including contacting learned societies, universities, research institutes and companies. Personally, I also bore people to death at parties and force my card on them if they are scientists or know any.
Because we’d done formative evaluation (i.e. talking to possible end users) as part of the development, we were able to focus communication and explain what the benefits would be to the scientists.
Once we’d run the pilot, word of mouth was one of our most powerful tools, as the scientists who took part really loved it. We also had evaluation evidence to back up our claims of the benefits.
To select the scientists (as we are now oversubscribed) we involved our end users. For the last event we took each scientist’s one-sentence description of their work and put it up blind (i.e. just their words, no information on age, gender, ethnicity, organisational affiliation) on a website where students and teachers rated each description. We, and a representative of the Wellcome Trust, also rated each scientist and we combined the scores.
We still want to find new ways of recruiting scientists, and particularly reaching the scientists we aren’t reaching at the moment. If you can help, let us know! And if you want to get involved, sign up here.
8. Teacher recruitment
Timeline for debate kit sign ups, kit 1
Again, used many routes. As you can see from this graph of sign ups for debate kits, some of the most powerful methods are direct mail, the Planet Science e-newsletter, and posting on the TES message board.
Also, for the big event in June, we got a significant number of sign ups from the STEMNET newsletter and a few from many other sources. We think it’s worth casting the net widely.
Again, word of mouth incredibly powerful. Especially because we recruited a ‘teacher panel’ of teachers to help us develop the project, before the pilot, and they became very invested in the project and really helped us a lot. This emphasis on consultation with teachers also meant we really were providing something they wanted, in a way that was useful to them.
Cross-marketing from our debate kits project was also incredibly useful. It enabled teachers to get a sense of where we are coming from, and the quality of what we provide, before making the commitment of signing up to a two week event.
We are oversubscribed with teachers too, and here part of our selection procedure is to deliberately pick as varied schools as possible (geographically, type of school). This has included Special Schools, Pupil Referral Units and Bristol Hospital Education Service. And also schools in Shetland, Northern Ireland and Oman.
We also try to get a representative sample results-wise. By which I mean, high-performing grammar schools are overly represented in the schools who apply, we try to even things out by picking more of the less academically achieving schools. We think it makes a bigger difference to those kids.
As with scientist recruitment, this is still a work in progress. If you can help us get the word out to teachers (taking part is FREE to schools), please do! Or if you are a teacher who would like to get involved, sign up here!
9. Evaluation strategy
Formative evaluation has been absolutely key. We started talking to scientists and science teachers before we did anything else. Asking them about what they wanted, what would work for them, what motivates them.
For teachers, we recruited a teacher panel of people interested in the project, who could give us instant feedback via email on everything from lesson plan ideas to terminology on the site. This also meant that they were invested in the project, when the pilot came around, and understood it and what they could do with it.
We also included young people, for example testing possible designs on them. And we made several school visits to observe science lessons.
We had a limited budget for the pilot, so much evaluation had to be done in house. But we set aside money to appoint an external evaluation consultant, Yvonne Harris, to spend a few days on the project. She advised us throughout, and also conducted some independent interviews with participants at the end, and audited our report. This was absolutely invaluable as she could check things like questionnaires and methodology as we went along, and suggest solutions we would never have thought of, and bring an independent perspective to the whole thing.
It also helped that we had grown the project organically, as we had developed over the years questionnaires that worked, and found (and corrected) many useability issues as we went along.
You can read our evaluation report on the pilot here.
We now have a much bigger budget for the roll out of the project, and we have devoted far more of that to external evaluation. Kate Pontin is now our external evaluator, and she has been invaluable in helping us think with clarity about what we need to find out and how we can do it. She has also been able to do far more schools observation than we can, as we are busy running the event while it’s on! This has been extremely useful.
Kate’s interim report will be unveiled at a special event at the Wellcome Trust on 20th October. This will be part of a special ‘Beyond Blogging’ event, curated by us. Tickets will be very limited, but do get in touch if you’re interested in attending.
10 Evaluation findings
Shane outlined some topline figures so far from the 2010 events.
TopLine Web stats for IAS June 2010
- 4,667 students
- 100 scientists
- 171 teachers
- 6,580 questions
- 3,085 comments
- 4,744 votes
More in-depth evaluation results will be published on 20th October, as above.
In the meantime, you can read our full evaluation report on the pilot here.
A summary of the pilot evaluation report here.
Or our short evaluation report on the 2009 event here.
11. Summary of the strengths and weaknesses of this format
Strengths
- Power reversal truly engages and empowers.
- Online gives access to scientists (for schools) and students (for scientists), without having to go anywhere.
- Doing it online also creates intimacy (makes it easier to ask real questions and break down barriers), compared to a scientist in person giving a talk.
- It also ‘levels the playing field’ – quieter and less confident students participate more.
- You’re reaching all the students in the class – not just the very keen, as you find in science clubs, etc.
- A conversation develops over time – over two weeks, students can read about the scientists, go away and find out more, ask questions, think about the answers, ask more questions…
- For teachers, the preparation work is done for them and they can concentrate on facilitating learning.
- For scientists, it saves time – no travelling to a school, they log in from their desk. Every moment spent participating is spent in engagement.
- Scientists also find the range and energy of the questions rejuvenating, thought-provoking and inspiring.
- And scientists get into the competitive aspect and have fun.
Weaknesses
- Some scientists (and some teachers) don’t like the informality of the project, although we think it has real value in making connections and breaking down barriers.
- We’ve disguised the learning and made it fun, so some teachers/scientists/students don’t see that it’s there and think it ‘won’t help them pass exams’. (God help us if that is the only thing some people think education is about).
- Schools IT (sigh!). Often school firewalls are over-enthusiastic and we do have some problems with schools blocking the site.
- It can be hard work for the scientists. Some had ~700 questions to answer!
- It’s expensive to run (although not compared to many other projects).
12. The Question Game
Our question comedy improv game! OK, so not very comedy, and not actually improv, but kind of a game, inspired by Whose Line is it Anyway? Audience members shout out a word, any word, and we search the site to see if there are questions (or answers) containing that word. Intended to give an insight into the enormous range of questions.
Here’s some results
If you want to play the question game for yourself (be warned, browsing the site can be addictive!), just go to the main page and type in the search box near the top.
Live chat
After the presentation, we wanted to give the audience a feel for what live chats are like, and why they are so popular with scientists, students and teachers. So we had a live online chat, with the audience taking the place of students. We gave out log in details and everyone in the audience who had a laptop (quite a few, it being Science Online) could log in and take part. We also showed the chat on the projector.
Some of the key benefits of live chats:-
- Access to people who couldn’t be there otherwise: We had a scientist in Michigan, one in Sydney, a teacher in Shetland, and a student who was in school. All of them had taken part in the event and answered questions from the audience about what it was like.
- Immediate and friendly
- Fun
- Newer comments appear at the top, so you need to read upwards.
- Discussion isn’t threaded – we’ve found that breaks up the chat too much and stops it being a communal experience.
- Chats can be difficult to follow at first, but you get used to it quickly. Students are often quite familiar with chatrooms and don’t find it’s a problem.
- There are two chatrooms side by side, one for students, one for the scientists. (In this chat that means one for audience and one for our participants). This makes it more difficult to understand an archived chat, however, we’ve found from experience that if the scientists and students are all in the same box, the scientists’ replies get lost in an avalanche of comments from students, so this works better.
Q+A
Despite our best intentions, Q+A actually ended up spilling into all of the session and we started taking questions during some of the earlier presentations, and during the live chat. Personally, I think this was in some ways the most engaging part of the session, as we were talking directly to people and they were asking questions, rather than us lecturing them presentation-style:-).
To get an idea of what was discussed, I think your best bet is checking the tweets from the session (link below).*
*We made a tactical error really: so that the twitterfall in our session was a useful backchannel, (rather than a tempting view of other parallel sessions that people were missing!), we suggested a special hashtag #iassolo. I now realise it would have been better to keep the #solo10 hashtag, but just add an extra #ias, then all the hashtags from our session would have been archived along with the others from the conference. Oh well, you live and learn.
Archive of tweets relating just to our session, hashtag #iassolo.
Students! Tell us what you thought about I’m a Scientist, and win a prize.
We're listening! Anatomy of the outer ear. Credit: Medical Art Service, Munich / Wellcome Images
This is a last reminder: We really want to hear from you!
- What do you think, did you like I’m a Scientist?
- What did you like/dislike about it?
- Was it better or worse than normal lessons?
You should fill in our survey and tell us what you think. It helps us make it better for next time. Also you’ll go into a prize draw for £20 WH Smiths vouchers.
Survey is here http://www.surveygizmo.com/s/307117/i-m-a-scientist-june-2010-tell-us-what-you-think-
Thanks for your help!
How DOES magnetism work?
He doesn't know either: Gilbert demonstrating the magnet before Queen Elizabeth /Wellcome Images
Last year I saw A C Grayling talk on happiness and it’s importance. He said that he bans his students from using the word ‘happy’, that it’s a lazy portmanteau word. He thinks that if you are forced to choose a different word – hopeful, exhilarated, content – you’ll think about what you really mean far more clearly.
I think the same can sometimes be true of jargon. Scientists taking part in I’m a Scientist have told us before that explaining yourself without using jargon is hard work, but unexpectedly rewarding. It means you have to think through what you mean and it exposes your mental shortcuts.
We saw a great example of this in Imaging Zone. A fairly innocuous-seeming question (Why do magnets attract and repel?) pretty much lead to the scientists realising that they don’t actually know how magnets worked. Not really. Not when they tried to actually explain it to other people, without using jargon.
Now these scientists include a man who spends many of his days working with an fMRI scanner, containing a magnet so powerful that you have to remove any ferrous object from your person before entering the room. But still, magnetism turned out to be one of those things that he learned about years ago, and sort of assumed he understood.
I’m expecting that some of you have the same feeling I did when reading that Q+A – ‘OMG, I don’t REALLY understand how magnets work either. How did I not notice that before?’ I think the thing is, most of us rarely discuss how magnets work. And when we do, we use technical words (‘dipole’, ‘electromagnetic’, ‘electron shell’) which we and our listeners all know, which can obscure the fact that you don’t truly understand the underlying mechanism.
By all accounts this question, and the attempts to answer it, lead to an awful lot of magnetism-related discussions at scientific breakfast tables and coffee machines around the country. So, one outcome, of just this question, has been much thinking about and discussing the mysteries of physics, by scientists, with colleagues, and others, about a subject they all thought they understood, but actually it turns out they’ve got lots of questions about it.
If the essence of science is asking questions and taking nothing for granted, then I’d call this a result.
This post started life as part of a mammoth post I’ve been writing about how June’s I’m a Scientist event went. The post has taken about three days so far and we’re up to 3,000 words. So I thought I should really break it into bits and start bunging some up now.
A scientist gives us the lowdown on what it’s like behind the scenes…
As you all know, there are evictions every day for the rest of the week! You get a vote every day, so remember to keep voting for your favourite scientist.
But how do the scientists feel, being bombarded with questions and then facing eviction? One of the scientists who took part in I’m a Scientist in March, the lovely Louise Buckley, has agreed to tell us about what the scientists taking part are REALLY thinking…
Louise Buckley
Hey there! I am Louise one of the scientists from the last “I’m a Scientist….” event. This competition is looking even more exciting than when I took part – I cannot believe how many questions are getting asked. It’s a total mash-up of questions – these scientists are going to need to take a holiday afterwards to recover!
I bet the scientists are starting to feel nervous now about who’s going to get evicted. We get labelled ‘the scientists’, but we are only human too. I know I was worried that you guys wouldn’t like me, that I’d get evicted because my science was crap or you thought I was a weirdo. Nobody likes to be evicted first. It’s kinda embarrassing. A bit like being a member of McKinley High’s Glee team 🙁
LMAO – my major fear was that you would ask me loads of questions that I didn’t know the answer too. I was terrified. And you did – and I coped and I learnt loads along the way. I can see from the questions asked at the moment that the scientists at the moment are getting asked loads of challenging questions. Keep ‘em coming. It reminds us that there is soooooo much more to science than our narrow fields of research. We sometimes forget that.
The scientists will be learning at the moment that it is okay to be wrong or not know the answers to every question. Sometimes you guys will know more than us, other times we will know more. You guys taught me that rainbows are circular – and I was like, nah, rubbish, wind-up, but it was true. Today’s scientists are learning all sorts of new things and they are learning them because of you.
But the most important thing that the scientists are learning at the moment is how to communicate our science to young people. Many of us don’t get that opportunity often. I know I was excited but scared that I wouldn’t be able to describe my work in a way that wasn’t total gobbledegook. These scientists are trying to manage it too.
We need to learn how to make our stuff sound interesting. So if you don’t think they have made something clear, ask and ask and ask again. Don’t “Yeah, checked out of this conversation a minute ago…” (sorry Sue Sylvester!) Give ‘em feedback, show interest, and you can make today’s scientists better. Remember they are all terrified of being evicted so DON’T be too tough on them personally – but DO give them a hard time scientifically. And that’s how Sue Louise sees it!
Busy, busy, busy!
Well the first week is almost over already and it’s been incredibly hectic. This is already the busiest event we’ve ever run – we’ve had nearly 4,000 visitors, just in the last few days!
Thank you to all the students for the interesting, funny and thought-provoking questions. And thanks to all the scientists for all the hard work they’ve put in answering them.
Students – feel free to comment wherever you want to on the site.
Visitors – although you can’t comment, you are very welcome to have a look round and read everything.
If you want to you can follow us on twitter – our twitter feed is http://twitter.com/imascientist, or the hashtag for the event is #IAS2010 – scientists and teachers, if you are on twitter, please use the hashtag if you are tweeting about the event, then people can find all the comments together.
We are having a whale of a time, we hope you are too. Best of luck everyone!
These scientists were just like real people!
Hi everyone, here’s a summary of how the March 2009 event went (PDF download). I was supposed to write a couple of pages so of course it’s 12 pages long (I’m the same with cooking – if I have a couple of friends round I make enough food for ten and we’re eating leftover curry ’til thursday…). Fortunately Shane (my boss) is busy being terribly important in London running his G20 bloggers tent, and hanging out with Bob Geldof (pictured), so he can’t tell me off about it.
Shane (just behind Sir Bob) at the G20Voice
What we found was pretty similar to the pilot (but it’s important to check everything is still working:-)). One of my favourite quotes was the one I used as the title. Students were over and over amazed that ‘scientists are just like real people’, and that they were taking time to talk to young people. They also couldn’t quite get over that we were giving them a say about something, “[I liked] being able to vote as a child and make a difference.” I find that quite moving – we all want to make a difference, don’t we? Do we give young people so few opportunities for doing that?
As for the grown-ups, all the scientists and all the teachers (who filled in the feedback survey) would take part again, and recommend the event to a colleague. Everyone enjoyed taking part, the scientists developed their communication skills and got inspired about public engagement, “I got a tremendous amount out of it, and I think I probably learnt a lot more from the students than they learnt from me!”, and the teachers felt their students benefitted enormously.
The key elements mentioned, once again, were:-
• Exciting and intimate medium for interaction
• Real scientists and real science
• Taking young people seriously and giving them actual decision-making power
• Supported by thought-provoking classroom discussion activities
But looking through all this data has confirmed for me again how important each of the elements is and how they work together.
For example, one thing that works is that live chats are an intimate and familiar medium for young people – so they can ask questions when they might normally be shy in class. But it’s the fact that they have a real decision to make about real science, primed by appropriate classroom discussion exercises, that gives some purpose to that conversation, and means it’s not just students asking what nipples are for.
We can’t claim this is entirely due to our genius as event producers (although I’m sure it’s a factor;-)), it’s also to do with the organic way I’m a Councillor, and then I’m a Scientist have developed*. We’ve seen what works and what doesn’t as we’ve gone along and built the event up. We also owe the biggest debt of gratitude to the teachers, scientists and young people who’ve helped us. Thank you everyone! I was going to talk about that lots more, but I have to go and catch a train to see my baby sister’s band play so it will have to wait for another time. Have a great weekend everyone!
*I particularly like how this theory confirms all the prejudices of my biologist worldview: Evolution, much better at engineering than engineers are.
modshane in Evaluation, IAS Event |
Tagged Evaluation, March 2009, thanks
Leave a comment
Supplementary questions
Lots of people have written back to me with comments on the draft plans for the teacher packs. Thanks everyone!
modshane in Evaluation, Science Engagement |
Tagged chemistry, data handling, Democs, development, draft plan, feedback, information sheets, lesson plans, teacher packs
2 Comments
Do you think scientific speed-dating would work?
Well we’ve been beavering away here in I’m a Scientist Towers. We’ve taken on a freelancer to develop the teacher packs. Becky Davies usually works at Techniquest Continue reading
modshane in Evaluation, How Science Works, School, Science Education |
Tagged Becky Davies, development, draft plan, feedback, information sheets, lesson plans, teacher packs, Techniquest
2 Comments
I’m a Scientist
‘I’m a Scientist, Get me out of here’ is a free online event where school students get to meet and interact with scientists. It’s a free X Factor-style competition between scientists, where the students are the judges. Students challenge the scientists over intense, fast-paced, online live CHATs. Then they ASK the scientists all the questions they want to, and VOTE for their favourite scientist to win a prize of £500 to communicate their work with the public.
Follow @imascientist
