Category Archives: Evaluation

Experiential public engagement training through I’m a Scientist

We often hear directly from scientists about the positive impacts they get from engaging with our enthusiastic and diverse audience of school students. We’ve been able to research these impacts in more depths as part of the ChallengeCPD@Bath project run by the University of Bath. This in turn is part of the Strategic Support to Expedite Embedding Public Engagement with Research (SEE-PER) project funded by UK Research and Innovation.

We looked at whether taking part helped scientists from the March 2018 event improve their communication skills and if so, explore why IAS worked as experiential training. The results showed scientists became more confident communicating with public audiences and helped them develop effective ways of talking about their work. The opportunity for frequent practise, honest student feedback, and the text based nature of interaction were among the elements highlighted by scientists as particularly helpful.

Using these results we can demonstrate I’m a Scientist is effective experiential training in public engagement, especially for scientists with less experience or lower confidence. We’ve already incorporated some of the evidence into new materials explaining the value in taking part. The strongest evidence currently comes from the qualitative research and we can look at ways of more effectively assessing the positive impacts through quantitative data.

Read more about the results and methods for the elements of the research:

  1. Scoring confidence in communicating
  2. Rating scientists’ sentences
  3. Follow-up interviews
  4. Methods

We’d like to thank Dr Helen Featherstone and the team at University of Bath Public Engagement Unit for the opportunity to be part of the ChallengeCPD@Bath project. Find more information on the other outcomes of the project at the Public Engagement Unit blog.

Taking part improves the confidence of scientists communicating their work to public audiences

Anecdotally, we often hear that taking part gives scientists a lot of confidence when talking about their work. We decided to 1.Ask scientists about their confidence before and after the event, and 2. See if tracking how they felt each day might reveal more evidence of experiential learning.

We asked scientists to score their confidence communicating with the public each day of the event. We hypothesised that once scientists started the event and had a go at actually talking to the students their confidence would dip, before increasing again as they became better at handling fast paced live chats and imaginative ASK questions.

Out of a possible 517 scores, the scientists submitted 414, with scores from 37 scientists submitted each day on average.

The distribution of final scores submitted by scientists compared to their earliest score, either from before, or in the first week of, the event.

Overall, there was a clear improvement in the scores submitted after the event. For example, 91% of scientists scored themselves at 4 or higher post-event and, in particular, 44% scored themselves at 5 (very confident) post-event, compared to just 13% beforehand.

Day to day, there wasn’t a dramatic dip then increase. However when split into two different segments, a clearer pattern could be discerned:

  1. Those less confident (3 or less) at the start grew the most as the event went on.
  2. Those more confident (4 or more) at the start became less so once the event started, before becoming more confident again in the following days.

Judging by the comments submitted with scores, it could be that Group 1 contained people who had done little public engagement beforehand, whereas Group 2 could include those who were more experienced.

Both groups finished with almost exactly the same score, suggesting the experience was beneficial no matter where you started. A couple of scientists also noted that being evicted had decreased their confidence slightly.

The scores also suggest women gain confidence at a faster rate than men. The confidence of female scientists increased on average by 27% during the event (3.52→4.45), compared to an increase of 10% for male scientists (3.76→4.13).

 

2. Quantifying skills improvement using one sentence descriptions is difficult

Every scientist who applies for IAS must submit a sentence that describes what they do. These are then judged by students and teachers and the average ratings are used to help guide who gets places in each event. We wanted to see if the ratings increased for the sentences written post-event, and if their readability had increased indicating that the scientists were better at communicating what they do. 

35 scientists submitted a post-event sentence to compare with their pre-event sentences.

Student/teacher ratings: The average for post-event sentences was 0.77, compared to a pre-event average of 0.90, representing a decrease in rating of 0.13

Readability: The average grade for post-event sentences was 9.23, compared to a pre-event grade of 10.32, representing an increase in readability of 1.09 grades.

The results do not suggest any clear conclusions. The decrease in student ratings, although indicating that the sentences were likely not as good, is very small. Likewise, the increase in readability is too small to infer impact. Subjectively, there is a sense that less exciting language has been chosen in the post-event sentences in favour of conciseness.

This could indicate either:

  1. Scientists do not become better communicators through the event
    • This is countered by hundreds of scientists reporting increases in their communication skills and continued use of effective phrases developed in the event.
  2. Other factor(s) influenced the test, for example, the motivation to impress the students was missing after the event had taken place.
    • If run again, we could provide a reward for the best rated post-event sentences to match the pre-event motivational condition. However, it may be hard to replicate the incentive provided by the opportunity to take part in IAS.

 

3. The format enables and encourages scientists to develop their communication skills

Hearing from the scientists interviewed, it was clear that all five believed taking part in I’m a Scientist had improved their skill in communicating with the public. Read the full narrative for each interview. 

In summary, benefits mentioned were:

  • Being able to better adapt their language to their audience (avoid jargon and technical language)
  • Developing succinct phrases and effective analogies to explain concepts that they continue to use
  • Getting better at explaining the wider relevance of their research to get people engaged, even if they had had lots of previous outreach experience.
  • Positive impacts on how they communicated in professional contexts.
  • More confidence when communicating, especially for those with less outreach experience. All had taken part in further outreach activities following the event in March.

The format of I’m a Scientist was particularly helpful for the scientists in the following ways:

  • Frequent practice – Daily chats and ASK questions over two weeks allowed for new ways of explaining to be tested out and phrases to be honed over time
  • ‘Two-way’ – Student feedback in the form of further questions let scientists gauge how well they were communicating
  • Online and text-based –  Students gave more honest responses than some scientists had experienced in person, scientists felt more comfortable answering challenging questions.
  • Shared experience –  Scientists could see and learn from how others in their zone approached answering the students

“…every time I was answering to the kids they would ask more and more and like that would sort of bring, break down the concept into basics and I understood that that is how you can explain your work to a non-specialist audience” Kezia, PhD Student, no previous outreach experience

“Some of the analogies I totally plagiarised from the students! They come up with their own ways of understanding what you do. You’d describe it and they’d say ‘oh, is that a bit like this?’ and you’d say ‘exactly’. And that’s really nice too, when they do the imaginative work of explaining it for you. I really liked that.” Max, PhD Student, some outreach experience.

“I think I’ve learned how to be more ‘less specific’, how to maybe explain more of the sort of bigger picture of where my research could go and then discuss into more details from there depending on if there’s interest. I think that helps people who don’t understand in technical terms more what I’m more what you’re doing if you sort of say well this is the end goal” Neil, Postdoctoral researcher, experienced in outreach

“It put in my mind that I can be asked any question outside my field that I don’t expect… I would do my best to answer those questions. And if I don’t really know something, I wouldn’t be afraid. I would say ‘Ah I’m sorry I don’t really know about this’ And I could say ‘What do you know about this?’” Walaa, PhD student, some outreach experience

“Now I feel I can, you know I’ve got that you’re in the lift and they’re getting out on the second floor speech….Because you’re not in person… there’s not that level of politeness, it’s just ‘I’m not interested anymore, bye, I’m going to speak to someone else’, rather than feeling like they’re stood in front of their teacher, or their mum and dad and feeling like they have to be polite to this person they are talking to…  I kind of think it makes you have to get better.” Aileen, PhD Student, experienced in outreach

 

4. Methods

Scoring confidence in communicating

A short survey was sent to every scientist on the Friday before the event asking them to score themselves from 1(Not very confident) to 5 (Very confident) in answer to the question ‘How confident are you feeling at this moment about communicating your work to a lay audience?’. There was also an open text field to record their thoughts on how they had arrived at that score.

The scientists then received the same question every weekday at 4pm during the event until the final Friday. This gave us a maximum of 11 responses for each scientist. Reminders were also sent first thing each morning in case they had missed the previous day’s email.

Rating scientists’ sentences

Students are asked ‘Should this person get a place in the next event?’ and then rates each sentence either Definitely not, Maybe not, Probably, Definitely! We then assign these options a rating from -2 to +2 to give an average rating for each sentence.

In March, we asked the participating scientists to write a new sentence describing their work after taking part and compared its average rating to that of the sentence they submitted in their application. Each sentence was rated an average of 18 times. We also processed the sentences individually through an online readability scorer using the Flesch-Kincaid test. In this test, the lower the score, or ‘grade level’ the more readable the text is.

Interviews

We carried out recorded phone interviews with a targeted sample of five of the scientists. We biased towards PhD students who have more to gain from training, and included people with varying levels of PE experience.

Posted on October 12, 2018 modantony in Evaluation, Scientist Benefits | Comments Off on Experiential public engagement training through I’m a Scientist

Scientist Interview Narratives

Here are narratives for each of the five interviews carried out as part of the I’m a Scientist element of the ChallengeCPD@Bath project, funded by UKRI. These interviews investigated how I’m a Scientist works as experiential training for scientists communicating their research. Read about the full report post on this research

Kezia – 1st year PhD student

I’m a Scientist was Kezia’s first outreach activity and she was certain it helped her become better at communicating her work. She says the experience gave her the confidence to do more communication with public audiences and she has started delivering workshop sessions related to her research for school students. Kezia also noted she has learned to adapt her language when talking to ‘non-specialists’. In particular, she now uses analogies to explain concepts related to her work, and says this approach came from her I’m a Scientist experience.

“It sort of made me realise that yeah, I am confident, yeah, I can do that (outreach activity)… I’ve also signed up for the same sessions to be done next year with the same schools.”

“It improved how to explain my work to the common audience so I’ve been able to talk about my work with even my non-scientist friends… I’ve been able to think of and come up with some sort of analogies when I’m explaining concepts to people, like I’ve never thought of that aspect before I took part in the I’m a Scientist contest”

What was it about IAS?

The iterative nature of I’m a Scientist helped Kezia adapt her language and develop analogies. For example, she says that the multiple follow up questions from students in response to her answers in live chats forced her to find more and more accessible language and new ways of explaining her work.

“...every time I was answering to the kids they would ask more and more and like that would sort of bring, break down the concept into basics and I understood that that is how you can explain your work to a non-specialist audience

Any changes in a work context?

Kezia not only feels she has become a better communicator in a professional context, but the improvement has been noticed by her colleagues.

“ I work in an interdisciplinary project so not everyone understands the technicalities of what I do so yeah it has actually had an impact on the way I communicate overall… I’ve got very good feedback from people unlike before so there has been some sort of influence.”

Anything else that stood out?

Having done no outreach previously, Kezia found the online format appealing as a ‘first step’.

“I’m from India so I hadn’t done any outreach back there so it’s just not the research culture over there, so it was a completely new thing to me so the IAS forum was an online forum where I felt a bit safe and confident”

 

Aileen – 1st year PhD student

Before I’m a Scientist, Aileen had had regular experience communicating her work to visitors at her research institute and at one off events like Big Bang Fair. She believes taking part helped her develop short ‘quick-fire’ explanations of her research, improved how she adapts explanations for non-scientists of all ages, and improved her ability to deal with unexpected questions

“…you know when somebody wants a ten second answer  of what do you do? Rather than going in to the whole ‘I do this this and this and I’m a PhD student blah blah blah it’s very easy when you do it all day to give them five minutes and not even scratch the surface whereas now i feel i can, you know i’ve got that you’re in the lift and they’re getting out on the second floor speech.”

What was it about IAS?

The need to regularly get across information ‘succinctly’ in live chats particularly helped Aileen develop her short explanations. She felt that because students could easily switch their interest to someone else, without worrying about being polite, she was forced to explain herself as clearly and quickly as possible. Aileen also appreciated the information about the class’s age or level before the chats to help her adjust to what they might know about already. The computer based, text only nature of the activity also meant she was comfortable taking a moment to compose answers to unexpected questions

“…Because you’re not in person… there’s not that level of politeness, it’s just ‘I’m not interested anymore bye, I’m going to speak to someone else’, rather than feeling like they’re stood in front of their teacher, or their mum and dad and feeling like they have to be polite to this person they are talking to…  I kind of think it makes you have to get better.”

“…It’s you know preparing you for the unexpected, it’s quite good and then you’re behind the screen so if you suddenly have 30 seconds when you clam up it’s a lot easier than when you’re standing in front of someone. So it’s quite good to practise it in that format, rather than ‘oh god there’s a major investor stood in front of me and I really need to answer them now.”

Plans for, or effects on, further public engagement

Brilliant Club – Aileen is looking to being more targeted with her outreach as she goes into the second year of her PhD (so an effect not related to her IAS experience) and preferring something where she can build relationships over a longer period of time.

Any changes in a work context?

Aileen believes her elevator pitch has helped her in professional contexts, such as conferences.

“in terms of turning up to conferences and that kind of thing, being able to explain what I do in a very short speech is helpful.”

Anything else that stood out?

Convenience of IAS a big plus for keeping her supervisors happy, perhaps pointing to the perception of outreach as ‘not real science’

“…I can do it over lunch  and I can still do, you know, maybe not the same amount of work that I would do on a normal day but I could still do 80% of the work that I could do normally …. and my supervisors don’t get upset about doing too much time doing outreach and not enough time doing my actual experiments.”

 

Max – 3rd year PhD student

Max is a 3rd Year PhD student who has previously written blogs and taught science at summer camps and in Oxfordshire. This outreach has not involved much of his own research and “that definitely sets I’m a Scientist aside, where the main focus was me and my work.”

By taking part Max says he has now developed a ‘better arsenal of metaphors and analogies’ for describing what he does, and feels he has a much better understanding of what public audiences find interesting about his work. He also feels more confident in talking about his work since taking part.

“…often you hear scientists being afraid of boring people or not being interesting, so you err on the side of not wanting to talk about your science at all… The experience showed me that actually, in general, people are interested in hearing about what you do, you just need to find the right way to engage their interest. And probably the way that you do that for a 14 year old is not that different to the way you do it for a 40 year old, especially if their knowledge of science is kind of comparable anyway.”

What was it about IAS?

Max thinks that the ‘two-way’ nature of I’m a Scientist was key to the impacts on his communication skills. Responses and feedback  from students directly helped him develop new analogies, and told him when he wasn’t explaining himself well enough.

“Some of the analogies I totally plagiarised from the students! They come up with their own ways of understanding what you do. You’d describe it and they’d say ‘oh, is that a bit like this?’ and you’d say ‘exactly’. And that’s really nice too, when they do the imaginative work of explaining it for you. I really liked that.”

“The whole thing was very two-way — you’re giving answers to the questions people are directly asking you, and then you get their responses as to whether or not they like the answers, both immediately because they’d say ‘that makes sense’ or they’d ask another question, and also long term because of the votes and the competition”

Plans for, or effects on, further public engagement

Max won his zone and plans to use his prize money to develop a podcast about his field. It will feature local school students asking questions about gene editing, and agrees that how I’m a Scientist lets the students lead the discussion while providing stimulus material (profiles etc) has influenced his approach.

“To me, that kind of two-way engagement where you’re actually having a scientific discussion, pitched at the right level, is just so much more productive than Brian Cox asking you to think about the universe or whatever.”

Any changes in a work context?

Max feels that answering the students’ ‘inventive’ questions has led to him talking much more with colleagues in the lab about their work and his own, and in a way that is more meaningful than technical.

“I think the amount that I chatted with my colleagues when I had these crazy inventive questions from students in I’m a Scientist, made me see that if I had that same level of discussion with them about my work — not just ‘how do I attach X to Y in this machine?’ but also ‘what does this actually mean?’ — it could be really productive. I think in general I’ve been talking to my colleagues more and been trying to see the wood for the trees a bit more and actually talk about the concepts, rather than just the nitty gritty, which as technical scientists we always get bogged down in.”

Anything else of note?

Max’s university, and others, approached him to write blogs posts about taking part in IAS.

Neil- Postdoctoral researcher

Neil does regular chemistry outreach as part of a tour of Scottish schools twice year, however it doesn’t involve talking about his own work. He says he saw a big change in how he communicated in both ASK and chat over the course of the event. He thinks taking part has improved his use of appropriate language and has changed how starts conversations with public audiences, now focusing first on the ‘bigger picture’ and the relevance of what he does to other people.

“I thought I was relatively good at communicating, obviously I still think I do, but I think my levels  of technical, what my perceived level of technical detail that people will understand and actual level may be a little bit different.”

“…now I know, I feel I know the level that I should be trying at and the level that you get more response from.”

“I think I’ve learned how to be more, more ‘less specific’, how to maybe explain more of the sort of bigger picture of where my research could go and then discuss into more details from there depending on if there’s interest. I think that helps people who don’t understand in technical terms more what I’m more what you’re doing if you sort of say well this is the end goal, rather than just saying we do this because it’s interesting or we do this because it’s an important piece of fundamental research that will lead on to more things. You sort of need to explain what the other things are and then come back abit to say why what you’;re doing fits into that”

What was it about IAS?

The back and forth conversation with students (or lack of, depending on his answers) helped Neil adjust the language he was using and the way he was explaining things, and the text only format was important for this to happen. He also learnt a lot by seeing how another scientist with lots of outreach experience was approaching their answers during live chats.

“If they’re asking me about my work or what I do day to day, erm, I think when I started answering them I think I was trying I tried to be a bit sort of over technical and not simplify enough and  I could see quite quickly that ‘hmm that’s not quite right so, cos you could see in the chats maybe you didn’t quite get more response’

I definitely learned quite a bit from especially from Lauren Webster, who’s a chemist from Dundee, cos she also does a lot of outreach at the moment. She used simpler language from the start … I found that if I sort of changed how i was answering things, maybe made things a little more vague, but more understandable, the conversations went on for longer”

“…text based, that’s how they’re used to communicating, it’s less formal and if you’re dealing with a number of people I’m guessing it would be quite hard to have a two way video conversation”

Plans for, or effects on, further public engagement

Neil thinks there was far more ‘interactivity’ with students in IAS compared to his schools workshops.

“…as soon as you’ve delivered the activity or the demonstration, you’re tidying up to do the next one and you’re thinking you know ‘right , i’ve got to go to the next place and get sorted again.’ This, there’s a lot more interactivity here and that was good.”

Anything else of note?

Neil enjoyed answering the science questions more than ones about himself outside of his work. He also found the chats with fewer scientists more enjoyable because it focused the conversation.

“…when there was only two or three of us (scientists) it was much easier to have thready conversations where both of us answering to the pupils and building on each others’ answers so I think we got more out of it when it wasn’t quite so chaotic”

 

Walaa – 1st Year PhD Student

Walaa had done a short talk competition last year and made the regional finals. After taking part in IAS she felt she had increased her confidence when dealing with unexpected questions, and learned how to provide more information for interested people. Walaa also feels she has gained a better understanding of what the public thinks and knows about the area of her research, bees.

It put in my mind that I can be asked any question outside my field that I don’t expect… I would do my best to answer those questions. And I would say, if i don’t really know something I wouldn’t be afraid. I would say ‘Ah I’m sorry I don’t really know about this’ And I could say ‘What do you know about this?’

What was it about IAS?

Walaa found the event was a good opportunity to practise explaining herself without technical language.

“I tried to start from the beginning, with very basic things, easy things. They don’t know any technical language so i tried to avoid using technical language, or if I have to use one then i would explain it, what does it mean , break it down into parts”

Plans for, or effects on, further public engagement

Walaa has since signed up as a STEM ambassador and delivered two workshops about bees

Any changes in a work context?

Doing IAS has been part of a process for Walaa about learning to sell herself

Anything else that stood out?

She feels the having IAS on her CV got her a job translating science writing into Arabic.

Posted on October 11, 2018 modantony in Evaluation, Scientist Benefits | Comments Off on Scientist Interview Narratives

Researcher Development Framework and I’m a Scientist

It’s well established that scientists doing I’m a Scientist gain just as many positive outcomes as the students they talk to. Using the Vitae Researcher Development Framework to frame these benefits allows us to articulate them in a way that resonates with universities.

Recently, we surveyed event alumni from centres of doctoral training (CDTs) to quantify the effect of taking part on relevant RDF descriptors. This is part of our ongoing strategy to build relationships with these institutions. In total, 37 alumni who had taken part in I’m a Scientist or I’m an Engineer at least 3 months prior responded.

Pleasingly, we saw positive effects across the board. Here we breakdown the key results by relevant RDF domain (don’t worry, we weren’t interested in all 60+ descriptors).

 

RDF Domain B – Personal effectiveness

81% of respondents considered taking part as contributing to their Continuing Professional Development (B3 Continuing professional development B3)

73% say that taking part increased their enthusiasm and passion for their research (B1 Enthusiasm, B1 Self-confidence, B1 Self-reflection)

  • Getting to reflect on their personal motivations and the interest that students showed in their research were the biggest contributing factor to this effect.

68% had seen benefits to either their professional networks, the number of opportunities offered to them, and their profile and reputation as a researcher. (B3 Continuing professional development B3, Responsiveness to opportunities, B3  Networking )

“It seems to have been a big positive on my CV, which I have to admit I didn’t expect. I was asked about it specifically both at interview, and later by my line manager during the induction process.” – CDT student

“It establish me as an enthusiastic researcher willing to engage with researchers. if future employers google me they find something positive straight away.” – CDT student

RDF Domain D –  Engagement, influence and impact

94% said taking part improved their ability to communicate research with public audiences

  • Being able to ‘adapt language for different ability levels’ was the most improved skill among respondents

92% said they adapted their approach to communicating with students as the event went on

91% continue to use phrases they developed during the event to explain what they do in other contexts (D2 Communication methods, D2 Communication media, D3 Public engagement, also A3 Argument construction)

69% said taking part had improved their understanding of the impact of their research on society in some way (D3 Society and culture, D3 Global citizenship, D3 Public engagement, also C1 Ethics, principles and sustainability)

  • The impact of their research on people’s everyday lives was the area most had improved this understanding. Understanding of ethical issues and the economic value of their research were other improved areas.

“Writing an answer, it’s much easier to stop using jargon, it was a good lesson in recalibrating yourself… Now when you go to do face to face outreach you feel more equipped.” – CDT student

‘”Engineers typically do not have a lot of time to give back to the society in the form of public discussion, but this experience highlighted that it is very important to have those conversations with new generations coming through and as a result I have participated in more public engagement activities than I previously would have.” – CDT student


Posted on May 15, 2018 modantony in Evaluation, Scientist Benefits | Comments Off on Researcher Development Framework and I’m a Scientist

Increasing students’ confidence with I’m a Scientist

“Many of our students lack confidence in their academic ability so they were buzzing when they realised they can hold their own in a conversation with intelligent, educated people and this helped them realise they are all scientists too!” – Julia Anderson, FE College Biology Lecturer


This general further education college is split across 3 sites in a large, post-industrial, non-university town. Students at the college took part in the I’m a Scientist Immune System and Genes Zones in March 2018. Julia tells us how meaningful engagement with the scientists increased her students’ confidence.

Why apply for I’m a Scientist?

One of the key things Julia is addressing with her students is their confidence issues: “Our students come to us lacking confidence. They’re doing A Level qualifications, but because they’re at FE college rather than a Sixth Form, they think they’re ‘2nd class’ and they really aren’t. I wanted to show them how worthy they are.”

Julia also wanted to get her students more involved in science so being able to connect them directly with scientists through the activity was another key incentive to apply: “They get unbridled access to scientists they wouldn’t normally get to talk to.”

What did the students do?

Students prepared by logging in to look around site and read scientists’ profiles. “We had a class discussion about how each scientist related to the A Level curriculum and debated the sort of questions students might want to ask ready for their live chat with the scientists.” Having access to the site for the full two weeks was helpful for Julia’s students to prepare and follow up either side of their live chat. “We finished with a de-brief lesson, talking about what was useful, what interested the students and their thoughts about the scientists as people.”

Growing students’ confidence

For Julia, the biggest benefit of the activity was improving her students’ confidence. “It was so good for them to see they can hold their own in a conversation with intelligent and educated people who have studied these topics for 10 years. They were buzzing and so excited to be talking to scientists and not sounding like idiots!” Taking part in I’m a Scientist not only helped Julia’s students gain confidence in their abilities, it helped them “realise they are all scientists too!”

Engaging students in a 2-way conversation

“Students and scientists were on fire, sending questions back and forth throughout the chats.” Julia explains how this 2-way interaction differs from a recent face-to-face where her students wouldn’t speak up; “we went to a university event and I was trying to get students to talk to the scientists and find out more about their work. They were too shy, saying things like ‘what if they don’t want to talk to me?’ There was none of that in the I’m a Scientist live chat. No fear from students that scientists would be too busy to talk to them, they were really comfortable.”

Improving social mobility

One of Julia’s students, from a deprived working class town he wishes to leave, was particularly interested in the background of the scientists; “the first thing he wanted to know was where the scientists went to school, was it state or private?” Participating scientists are selected to ensure a range of backgrounds and routes into science are represented. Julia’s student initially switched off from one scientist who attended private school but soon changed his mind, “when the scientist explained that the majority of students on his undergraduate course were from state schools, my student could relate to this academic route.” Julia managed to talk this student out of leaving college part way through his qualifications “I felt if he did, he would never leave the town he wants to leave. The scientists in this activity had varied backgrounds and he was content with that, it helped him see you can grow up working class, on a low income and get ahead.”


To help your students gain confidence in their abilities through I’m a Scientist activities, register your interest at imascientist.org.uk/teachers or contact admin@imascientist.org.uk for more information.

Already registered? Don’t forget to apply for the next event – we email registered teachers when applications open (about 2 months before the event starts).

Posted on May 14, 2018 modkatie in Case Study, Evaluation, News, School, Teachers, Widening Participation | Comments Off on Increasing students’ confidence with I’m a Scientist

Using I’m a Scientist to increase participation in higher education

“Traditionally, not many of our students go on to university. I’m a Scientist helps by allowing students to relate to scientists and helping them see the value of studying at a higher level.” – Mark McNally, Science Teacher

 


A mixed 2-19 academy, where over two thirds of the school population are students from disadvantaged backgrounds, took part in I’m a Scientist in March 2018. Mark tells us how the activity helped interest his students in science careers and consider higher education.

Why apply for I’m a Scientist?

Despite a good attitude to learning among the students, not many go on to higher education; something the school is working to improve. Mark explains how he wanted to challenge his students’ preconceptions of scientists and help them consider studying science at a higher level; “I wanted to show our students science is not just for ‘weird people with crazy hair and lab coats’ and help them find interest in things going on now in science to increase the chance of them pursuing STEM subjects in the future.”

What did the students do?

The activity was covered in three lessons led by the class teacher across a two week period. Students started by considering how to judge the competing scientists, then got to know them using their profiles and asking questions on the site. The final lesson involved an online chat where students typed their questions and responses to scientists in real time before voting for their favourite scientist.

Did it work?

Mark agrees that I’m a Scientist helps raise students’ science capital, increasing the likelihood of them studying STEM subjects or using science in their future professions. “Raising awareness of a variety of careers they can go into and getting them in contact with scientists helps students see it’s something they could do.”

Opening students’ minds to higher education

I’m a Scientist provided an opportunity for Mark’s students to engage with science professionals who have studied at a high level, helping to open students’ minds to options they may not have otherwise considered. “I think one of the main barriers for our students is that not many come from families with an academic background so they don’t often consider academic routes,” explains Mark, “through this activity, my students connected with academic people and found out about things that interest them in terms of a future career, so they are more likely to pursue an academic route.”

”Even if students don’t want a career in STEM, they can now see the value of studying at a higher level and if they do want a career in STEM, this activity helped cement that for them.”

Satisfying quieter students’ curiosity

Mark’s students developed knowledge and understanding on current scientific topics, “I’m a Scientist allowed the students to engage with the kind of science that’s going on right now in the world.” Mark also told us of the importance for his students to be able to ask whatever they liked throughout the activity, “for a lot of students, I’m a Scientist was about satisfying their curiosity. It’s important because they have so many questions and they don’t always ask, especially the quieter students, but using this platform allows them to get their questions addressed.”


To support your students in considering higher education through I’m a Scientist activities, register your interest here: imascientist.org.uk/teachers or contact admin@imascientist.org.uk for more information.

Already registered? Don’t forget to apply for the next event – we email registered teachers when applications open (about 2 months before the event starts).

Posted on May 2, 2018 modkatie in Case Study, Evaluation, News, Teachers, Widening Participation | Comments Off on Using I’m a Scientist to increase participation in higher education

Providing STEM opportunities for distant schools

“As we are a remote rural community we do not have a huge variety of careers on our doorstep but these events help to bring them closer to pupils. More students should be getting these funded opportunities across the UK.” – Emily Tulloch, Science teacher on the island of Unst.


The most northerly school in the UK is located in one of our most distant areas in the Shetland Isles. Emily tells us how I’m a Scientist allowed her remote students to explore a range of STEM careers and increased motivation to learn science.

Why apply for I’m a Scientist?

“For me, it’s all about increasing Science Capital.” Emily often tries to provide STEM opportunities in school, particularly to help students broaden their understanding of careers, but it can be a challenge. “The STEM ambassador programme is great and we have a number of ambassadors based in Shetland. However there are significant barriers for them to reach the school – it can take almost a day to visit for a 1 hour talk and of course there’s the cost implication too.”

The school often rely on parents to demonstrate different careers to students “we’re lucky that our community is so supportive and parents working in STEM are very willing to visit the school, although the variety of careers is still limited on our island”. Emily wanted to use I’m a Scientist to increase the range of STEM roles her students find out about.

What did the students do?

The 17 students in S1-3 (Year 7-9) took part in the Molecule Zone this March. They read scientists’ profiles and posted questions to scientists on the site throughout the 2 weeks. Students also took part in a 30 minute live chat session where they typed their questions and responses to scientists online in real time.

Emily used the activity to connect to curriculum areas and the careers work going on in school. “It’s a broad activity so easy to link to something we’re doing or have just covered in class. I also tie I’m a Scientist into My World of Work, which our students access to find out where they could study the course or the career steps required for their preferred role.”

Did it work?

Being online, I’m a Scientist provided a chance for Emily’s students to connect with scientists in a large variety of roles, generating interest in STEM careers without the need to travel. “They see different careers we don’t have in the local area by chatting with scientists all across the globe…There are lots of young people who live in remote areas compared to the rest of UK and they should all get same opportunities; this activity allowed me to provide this at no cost to the school.”

Emily also told us how involvement in the activity has improved science learning in her classes “It’s engaging and stimulating for students to actually speak to live scientists doing real-life science and discuss what they think might happen in the future. When students are engaged and start to see the real life impact of a subject, they’re more enthusiastic about it – I noticed an increased motivation to learn following I’m a Scientist.”

What else did students gain?

Asking questions to the scientists allowed Emily’s students to “develop essential literacy and communication skills, particularly in the live chat where they have to consider how to engage in a group conversation” whilst “researching the scientists helped them learn how to find information online.”

Emily also commented on how the activity is good for students’ health and wellbeing, by using the internet chats as a relevant educational tool “students like communicating in this way online and this activity helps promote positive online communication and staying safe online, rather than fighting against social media and similar technology in school.”

What would you say to a teacher who is hesitant to take part?

“Give it a go – it’s amazing and you don’t realise the full impact before you take part but afterwards, you realise how hugely beneficial it is, especially for students in rural communities. The whole process has knock on effects for your science teaching as students are more engaged. It’s very stimulating for science and every pupil should have access to it.”


Distant (under-served) schools are given priority places in I’m a Scientist. To support your students in exploring a range of STEM careers with I’m a Scientist activities, register your interest here: imascientist.org.uk/teachers or contact katie@mangorol.la for more information.

Already registered? Don’t forget to apply for the next event – we email registered teachers when applications open (about 2 months before the event starts).


 

Posted on April 6, 2018 modkatie in Case Study, Evaluation, News, Teachers, Widening Participation | Comments Off on Providing STEM opportunities for distant schools

Broadening horizons for students in a deprived area

“It was a really easy project to engage with for both children and teachers; it only took me about 30 minutes to prepare for all 3 lessons. I was pleasantly surprised with just how excited students were to get responses from serious adults about their work and the adults’ interests.” – Vicky Heslop, Year 6 teacher


A junior school that meets our widening participation criteria took part in the Climate Zone of I’m a Scientist for the first time in March 2018 with their three Year 6 classes. The activity broadened student aspirations, improved enquiry skills and challenged their perceptions of scientists.

Why apply for I’m a Scientist?

The school is in a small town with low levels of aspiration and social mobility where very few young people go on to higher education. 47.5% of students at the school qualify for Pupil Premium funding and the proportion of free school meals eligible students is over twice the national average.

Teachers were finding it difficult to promote working scientifically skills and to support students in developing an enquiring attitude within the curriculum. “We wanted to provide a broader experience of what science is and to increase student aspirations,” says Vicky.

What did students do?

The activity was spread across 3 lessons on different days which Vicky felt was “particularly good for the anxious students in Year 6, allowing them to ‘take a break’ from SATs preparations with something that was still a valuable use of time.” Students started by considering how to judge the scientists, then got to know them using their profiles and asking questions on the site. The final lesson involved an online chat where students typed their questions and responses to scientists online in real time. “Connecting with the scientists online provided an opportunity to have positive social contact with adults in roles they wouldn’t normally have contact with,” observed Vicky.

Did it work?

Vicky told us science has become more relevant and attainable for students and how their aspirations have broadened to include science – “After the live chat, students were telling me how they’d like to become scientists.”

Students’ perceptions of scientists have changed as a result of taking part – “I thought scientists were boring but now I think they’re AWESOME!” – Year 6 student

What else did students gain?

“Students developed oracy skills,” says Vicky “and the ability to ask appropriate questions.” Vicky also explained how the school has had issues with inappropriate use of instant messaging and how I’m a Scientist was “a great way to demonstrate a positive use of this technology,” helping students learn appropriate online etiquette.

When asked if she’ll take part next year, Vicky says “I hope to and I’ll be telling the other teachers about the activity as it’s such a good one for our students.”


If you’d like to broaden your students’ aspirations with I’m a Scientist activities, register your interest here: imascientist.org.uk/teachers

Already registered? Don’t forget to apply for the next event – we email registered teachers when applications open (about 2 months before the event starts).

 

Posted on March 27, 2018 modkatie in Case Study, Evaluation, Teachers, Widening Participation | Comments Off on Broadening horizons for students in a deprived area

School engagement in STEM enrichment: Effect of school location

In recent years funders of public engagement and outreach activities have made a priority of reaching underserved audiences.

Wherever we looked we found anecdotal evidence that while, as a sector we were becoming increasingly effective at reaching schools in deprived parts of our metropolitan areas, rural communities continued to miss out.

But anecdotal data only gets you so far. We wanted to find out just how much the more remote schools were missing out. We also wanted to know what constitutes a remote school in this context.

First we looked to see what information already existed. There are some organisations who hold vast quantities of data about scientists and engineers visiting schools. However that data was not easily available for analysis. So we turned to a source we could access. The teachers who have signed up for our projects.

We wanted to find out:

  • Whether some schools access more STEM enrichment activities than others
  • And if so, is the location of the school a limiting factor
  • Whether there’s a difference between visits to the school, and visits from the school

Method:

We surveyed teachers who have registered for any of our UK based projects, including I’m a Scientist, I’m an Engineer and Debate Kits. The teachers were predominantly subject teachers. We therefore worked on the assumption that teachers can best talk about the classes they teach, and it would be unreliable for them to make assumptions about the wider school.

Who answered the survey?

Survey respondents: Subject taught (left), type of school taught at (centre), and location of schools (right).

What did we find?

Distance really matters.

Access to STEM engagement is not universal. Schools within 15 minutes drive of a major research HEI are twice as likely to get a visit from a university scientist than those over a 30 minute drive.

Interestingly, there doesn’t seem to be a difference when arranging to take students off-site for visits, but with teachers citing costs and time restraints as barriers to offering these activities, it’s clear to us that we need to be improving the offer to more distant schools.

If you have any comments, thoughts, or would like to know more, please get in touch with shane@mangorol.la.

Posted on November 30, 2017 modnaomi in Evaluation, News, Widening Participation | Comments Off on School engagement in STEM enrichment: Effect of school location

Thinking about Science Capital

We’re thinking increasingly about Science Capital and how we apply it to our projects. It is a powerful concept that resonates strongly with what we aim to do and we want to make sure our projects make as much of a positive contribution to young people’s Science Capital as they can. We are looking for ways to evaluate in relation to Science Capital to show whether we are achieving this.

What is Science Capital?

If you’ve read the 2013 ASPIRES report you’ll be familiar with this striking graph showing that although nearly 80% of UK students value science, less than 20% aspire to be scientists. Why is this?

Graph courtesy of ASPIRES/ASPIRES2: ucl.ac.uk/ioe/departments-centres/departments/education-practice-and-society/aspires

Based on their research, the ASPIRES team have developed the concept of Science Capital; the combination of experiences, personal connections, knowledge and attitudes that contribute to how much a young person identifies as a “science person”. The researchers found that young people with high Science Capital are more likely to have science-related career aspirations than those with lower Science Capital.

Dimensions of Science Capital

The ASPIRES researchers identified eight dimensions of Science Capital. We’ve summarised them below. For more detail follow the link to the Enterprising Science site.

  1. Scientific Literacy
    A young person’s knowledge and understanding about science and how science works. This also includes their confidence in feeling they know about science.
  1. Science-related attitudes, values and dispositions
    The extent to which a young person sees science as relevant to everyday life.
  1. Knowledge about the transferability of science
    Understanding that science qualifications, knowledge and skills have broad applications and are useful for a wide range of jobs beyond and not just in science.
  1. Science media consumption
    Engaging with science content in the media (books, television, or on the internet).
  1. Participation in out-of-school science learning contexts
    Participation in, e.g. science museums, science clubs, fairs, etc.
  1. Family science skills, knowledge and qualifications
    Having family members with science qualifications, skills, and interests.
  1. Knowing people in science-related roles
    Knowing people in their community who work in science-related roles.
  1. Talking about science in everyday life
    How often a young person talks about science out of school and the extent to which they are encouraged to continue with science.

Science Capital and I’m a Scientist

Our data and experience suggest that taking part in our I’m a … events has a measurable impact on students’ attitudes towards science and make a positive contribution to young people’s Science Capital.

We’ve looked at the 12 dimensions of Science Capital listed above and thought about whether and how I’m a Scientist contributes to each. There are three dimensions where we think our contribution is – or should be – most significant: knowing people in science-related jobs, scientific literacy, and knowledge about the transferability of science.

Knowing people in science-related jobs

Description: The people a young person knows (in a meaningful way) in their family, friends, peers and community circles who work in science-related roles.

We had a lot of discussion about whether the interactions between young people and scientists during I’m a Scientist, qualify as knowing someone “in a meaningful way”. Ultimately, we felt that while our events don’t fully fit the description, they fit the spirit, and – we think – make a small, though convincing contribution.

In fact, this is the area where we think our projects are most distinctive among STEM interventions. Young people who take part, engage in sustained and enthusiastic interactions with a group of real STEM professionals. Some students even form opposing teams to support their favourite scientist!

How we contribute:

  • Through voting, young people…
    • Take time to consider what is important in making a good scientist
    • Learn about scientists by reading their profiles and deciding who to vote for
    • Make personal judgements based on their direct interactions with the scientists and choose favourites
  • Through ASK and CHAT, young people…
    • Hear about scientists’ motivations and achievements
    • Learn about scientists’ interests, likes and dislikes, and find areas in common
    • Hear about life as a scientist from scientists’ points of view
  • Through the whole process, young people learn that many different types of people become scientists

Scientific literacy

Description: A young person’s knowledge and understanding about science and how science works. This also includes their confidence in feeling that they know about science.

We see plenty of evidence that suggests we contribute to this. Students ask scientists and engineers a huge variety of questions about science and how science works. In addition, we hope that the student-led approach and the willingness of scientists and engineers to answer pretty much any question gives students more confidence that they know about science.

How we contribute:

  • Scientists answer questions on scientific topics and students learn from their answers
  • Students learn about scientific process, ethics, and science in society
  • Young people develop their confidence in feeling that they know about science as they ask questions

Knowledge about the transferability of science

Description: Understanding the utility and broad application of science qualifications, knowledge and skills used in science (e.g. that these can lead to a wide range of jobs beyond, not just in, science fields).

Knowledge about the transferability of science is a really important dimension, and one where we should be able to make a contribution. At the moment, we are concerned we’re not getting it quite right.

We have run the occasional zone with candidates who have a science background but work in other roles, but in general zones tend to comprise only practicing scientists or engineers. We’re concerned this reinforces the “science = scientist” idea rather than helping students see that science skills are transferrable.

We’re working on evaluating whether our concerns are justified, and how we could adjust the composition of future zones to improve things.

Next steps

We’re just at the start of this process of evaluating projects in relation to Science Capital. As mentioned above, the process has already made us think about how our I’m a… projects contribute to young people’s knowledge about the transferability of science.

We’d really welcome discussion about this to help us develop our thinking so please get in touch and share your thoughts, either in the comments below, or elsewhere.

As a final note, there are also a few Science Capital dimensions where we don’t have much influence, and one – science-related attitudes, values and dispositions – where we think we do contribute, but would find it hard to evidence.

Posted on October 6, 2017 modbecky in Evaluation, News | Comments Off on Thinking about Science Capital

How do students from different schools engage with IAS?

Thousands of school students meet scientists through I’m a Scientist every year, and they ask thousands of questions.

In June 2017, over 3,000 students took part, asking scientists more than 2,500 questions in the ASK section alone. This is also the event that we implemented our question coding system across all the zones to see what students are asking about.

This all got us thinking:

Do students from different types of schools ask more or less of certain question types?

We’ve identified two groups we want to look at:

Under-served: Schools more than 30 minutes travel time from a major research HEI

Widening Participation: Schools with an above average number of students eligible for free school meals

Taking the questions from the I’m a Scientist zones in June 2017 it appears that:

  • Overall, the split of questions is similar across all groups of students
  • Under-served students ask more “science topics” and “personal” questions, getting to know the scientists outside work
  • WP students ask slightly more questions around “careers and education”

 

 

 

 

 

 

For all groups, questions about careers and education are the most common, and questions about the event or completely random and unrelated are the least (phew).

But why the differences?

Are they indicative of how students at different schools view scientists and STEM?

Do under-served students ask slightly more “personal”, and science topic questions to compensate for fewer opportunities to meet scientists in person?

Does the slightly higher percentage of WP students asking about “careers and education” demonstrate their greater interest in understanding future opportunities in STEM?

 

We don’t have the answers, but it’s certainly interesting to us that there are small differences in the types of questions students’ ask.

If you have thoughts on why this might be, we’d love to hear them, or if you want to talk about how you can support more under-served and WP students to have this opportunity get in touch: shane@mangorol.la | 01225 326892

Posted on September 14, 2017 modnaomi in Evaluation, News, Widening Participation | Comments Off on How do students from different schools engage with IAS?

“I’m a Scientist is great, but wouldn’t it be better if students could see and hear the scientists too?”

This is a question we get asked from time to time. Here we explain why we’re confident that text interaction remains the best format for effective online engagement: it makes students and scientists more comfortable, levels the playing field between adults and children, and makes the events accessible to a wider audience.

Students are more familiar with text-based chats.

There is growing evidence¹ that young people communicate most via text and less and less through phone or video and we’re hearing that anecdotally too. Feedback from teachers has pointed out that students are not only more familiar with a text format but also more comfortable with it.

All parties feel more confident about not being visible.

As an ex-teacher myself, the thought of making a class visible online to an unknown person via a webcam makes me uncomfortable. I’d also be concerned about scientists inadvertently displaying confidential or inappropriate material in the background. Text-based chats make it much easier to protect student identity and safeguard young people online.

One teacher told us his students didn’t want to be involved in I’m a Scientist at first because they assumed it would involve webcams and audio. Once he showed the class what the text-based chat looked like, they were much more comfortable and keen to get involved. Consequently these 16 students, who wouldn’t have participated in a video chat, asked 89 questions in their text-based live chat.

Both students and scientists feel less exposed through text-based chats in comparison to audio-visual. For students, this has a huge impact because the fact they are “hidden” gives them the confidence to fully engage with the event and students who are often too shy to speak up in class are able to do so in this environment.

A big part of what we do is break down stereotypes and whilst scientists have a profile picture on the site, the lack of video and sound means students focus less on what the scientists look and sound like and more on what they have to say.

Being text-based also provides an opportunity for scientists to communicate clearly with the students and provide meaningful responses:

Logistically, it’s easier.

A number of scientists have commented on the convenience of text-based chats. With our current model, scientists can take part from their office or usual work environment; having audio-visual chats would make these kind of outreach activities inaccessible to some scientists.

In fact, our current model is so easy to access, scientists have taken part in public places when out of the office, including a motorway service station and Glastonbury festival!

Online outreach activities, both text-based and audio-visual, allow schools to connect with scientists despite being in very different locations; this provides a school with access to hundreds of STEM professionals across a range of roles. Online outreach is especially important for rural schools, but it only works if the schools have the infrastructure to facilitate the projects. A number of our rural schools have previously expressed concerns about bandwidth. Whilst their internet connections are able to cope with text interactions, a video chat would be problematic in these locations.

Our chats are fast-paced but everyone gets their say.

Whilst it is possible to create a video chat with multiple scientists, turn taking would need to take place, losing the fast pace that we currently have in our text-based chats. This means fewer questions answered in the same time frame and less impactful engagement.

A similar challenge occurs when considering how students ask their questions. Undoubtedly the teacher would need to facilitate the session to decide who speaks next and, potentially, which questions should be asked. This intervention would diminish the excitement element of I’m a Scientist. It would also dissolve the direct link between student and scientist in our events, taking the ownership away from students.

Through our text-based chats, we tackle the ‘those who shout loudest get heard’ issue with chat features and moderators creating a situation where all students get attention. The number of replies a student gets is displayed to scientists and moderators who can then focus on students with fewer replies.

Our events are about more than the live chat.

Through our text-based chats and the two week long events, students not only have time to fully engage and ask questions to the scientists, they also have the chance to build a rapport with the scientists and get a true picture of what their lives entail. Students have access to five experts, providing more breadth and an opportunity to interest more students in the class.

The competition element of our events generates excitement as the students are in charge of who wins and the scientists are fighting for student votes. Much of our feedback from teachers explains how this creates longer-term student engagement than a one off chat. Students continue to talk about I’m a Scientist long after the event finishes – an indicator of the popularity and success of our events.

We could go on but the main point is…

To get the most from outreach opportunities and maximise engagement of all parties, it is essential that students and scientists feel equally comfortable and the format is as accessible as possible. We’re confident that our current text-based chat model is the best way to do this.

If you would like to talk more about our reasons, leave a comment or email me at katie@mangorol.la, we’d love to hear what you think.


¹A 2014 Ofcom report found that children aged 12-15 spend over 50% of their communication time on text messages, compared to just 6% on phone calls and video calls combined.

Back to top

Posted on August 4, 2017 modkatie in Evaluation, IAS Event, News, Science Engagement | Comments Off on “I’m a Scientist is great, but wouldn’t it be better if students could see and hear the scientists too?”

What do students ASK about?


Our events generate huge amounts of interesting data, which we know contains all sorts of valuable insights. When there’s so much of it, it can take a while to work out how to make best use of it. 
One example of this is thematic analysis of the questions students post in ASK. Over the years of running the event we’ve developed a pretty good sense of the types of things students ask about and will always showcase great examples of questions. Converting that into a systematised, reportable analysis is more difficult, but can be done. So, we have developed a system for coding questions based on a set of themes that students often ask about, and after some trials in March, we’ve applied it to all zones in the June event (read the latest reports here).

At their broadest level, most student questions fall into one of three areas: questions about the science; questions about the scientist; questions about the I’m a… event. But we can look at this in more detail, considering elements of science capital (look out for a separate blog on this coming soon), and “how science works” (although this term has now been re-framed as “working scientifically” in the latest UK curriculum, we think it is still descriptive and useful for us). We’ve identified five top-level themes that we will report on. Each of which has one or more sub-themes which we are using to code the questions.

 

Science topics

covers facts, theories and knowledge about science. These can be related to the scientists work, or more general questions about science topics. Examples include…

When was the first epidemic discovered?

What animal has the worst venom and what does it do?

Can you give a robot human emotions?


How science works

looks at the process, motivations and ethics behind science. This could be finding out more about how decisions are made, why people chose science, or looking at how science fits into wider society. Examples include…

Is your work accurate and can be depended on?

Why do you think that a lot of people believe that psychology isn’t a science?

How did you come up with the idea of making MRI scans faster?


Career and education

includes the experiences of the scientists and aspirations of the students. Questions in this category build a greater understanding of real-life careers and education, including previous experience, current situation and future plans. Examples include…

What criteria did you use to choose your university?

Do you think you’ll ever leave the science job route for something different?

What is a PhD and how do you get one?


Personal

encompasses questions about scientists lives, knowledge and opinions outside of their work. Examples include….

Do you still have a social life?

From studying human relationships have you altered how you interact with people in day to day life?

What’s your opinion on the election?


Event/other

includes questions about the event and prize money, as well as a handful of questions that really don’t fit anywhere else. Examples include…

Do you think you can win I’m a Scientist?


Method

Using moderators and the event team each zone is coded, and then checked, so each set of questions have at least two run throughs. We use a coding guide as reference for consistency, and we are always building on our bank of reference questions for each category.

Students can ask anything they want, so it’s not always a clear-cut decision. Some questions overlap the different categories, but in general fit best within one of the reportable themes. There are some trickier questions that a few people will make a decision on together, and when we see a lot of similar questions we can agree how to code these. Although this won’t eliminate the ambiguity, it will help us to be consistent and transparent in how we code.

 

Results

Here’s how the questions distribute across all I’m a Scientist zones for the June 2017 events.

 

You can see the outcomes for individual zones in the latest I’m a Scientist reports and I’m an Engineer reportsWe’re going to keep refining this coding to make it more consistent and the best reflection of our events it can be.

If you have any comments, thoughts, or would like to know more, please get in touch.

Posted on July 17, 2017 modbecky in Evaluation, News | 1 Comment

Gender differences in online engagement

“I thought scientists just looked like they do in the film Flubber and experimented on aliens or weird stuff but when I found out you liked Taylor Swift I realised you are more down to earth and not like mad scientist :)” – Student, November 2015

I’m a Scientist is about connecting pupils with real scientists. Something we want to find out is what effect this interaction has on different groups of students. Recent research by the Institute of Physics has done just that.

The IOP has carried out an independent research project to gain an insight into the behaviour and attitudes of boys and girls who do I’m a Scientist. As part of their Improving Gender Balance project, the IOP funded two zones: Terbium Zone and Osmium Zone.

The researchers anonymously surveyed the students’ attitudes to science and scientists before and after the event. We also provided them with all the student’s interaction data from the sites. Every line of live chat, every question in ASK. Specifically the researchers wanted to look for gender differences in two areas:

  1. How taking part in the event changed students’ attitudes to science and scientists
  2. The types of questions that students asked the scientists.

The first results from analysis of this treasure trove are now online at the IOP blog.

IOP gender difference Q1

Physicist, know thyself: After the event, girls in particular reduce the negative words they use to describe you.

Check out the post for their full findings on the effects of taking part in I’m a Scientist, including:

  • positive changes in word choice to describe scientists
  • an increase in girls’ awareness of careers where Physics A-Level is useful
  • girls’ improved confidence in talking to physicists

Girls want to see the scientists as real people

The report also dissects what girls and boys want to know from the scientists. Interestingly, as well as asking more questions overall, girls seem to ask many more personal questions, asking more than boys about job satisfaction, the career goals and achievements of the scientists, and their motivation to work in science.

For us, the identified trends make sense. For example, a boisterous classroom culture can be a barrier for quieter children to engage with visitors. As Natasha at the IOP notes, ‘the online, anonymous nature of the live chats gives students more freedom to ask questions than a traditional careers talk or even a speed-networking-style careers event.’ Girls, as a group, seem to relish this freedom.

And we believe that for science to be appealing there’s no need to glam it up with explosions and ‘wonder’. The more students hear about the reality of science from people like them, the more positive they feel about it. This research backs up that view.

Research with us

If girls are a group that benefits from this increased exposure to scientists as real people, it’s likely other underrepresented groups do too.

I’m a Scientist provides a unique way to study the ways children interact with scientists. We’d love to see more research being done using our data. If you think you, or someone you know, might be interested in analysing what hundreds of children want to know about science, just drop me a line at antony@mangorol.la

Read more on the findings in the full blog from the IOP: Online event gives insight into gender imbalances

And find out more about how Osmium Zone was set up: I’m a Scientist, not just for scientists

Posted on October 21, 2016 modantony in Evaluation, News | Comments Off on Gender differences in online engagement

Careers Zone

Report Cover

Careers Zone report, click to download

The Careers Zone is a new zone where students attending a careers fair at their school can talk online with a range of scientists and engineers, based all over the UK.

A lot of the time, school careers fairs are only able to accommodate businesses and organisations in the local area. We want to give students an opportunity to ask their questions to experts working in a multitude of areas, letting them see how diverse STEM careers can be. In turn, STEM professionals can pass on honest advice and information, showing students what it’s like to be a real engineer or scientist.

We selected a range of experts to take part, including a deep-sea researcher, a scientist for the NHS and a Jaguar Land Rover engineer. There were also engineers who had gotten into their professions through an apprenticeship. It was important for us to let students see the different options they have as they make decisions about their future careers. Experts could log in to the site from wherever they were for their scheduled Live Chat sessions, and then answer questions sent to them as they had time throughout the day (we even had an engineer take part in a chat from Mexico).

We set up on a long table with iPads and laptops for students to sit down at and use. We had two banners which showed some example questions, but often students came with a specific area of interest and asked us who the best expert was for them to talk with.

After two pilots, we’ve learnt it’s important for us to be on the same level as all the other stands at the fair, making approachable for students to ask who we are, and whether they can join in. We were also told that there was an announcement in the school newsletter about us being at the fair, and students had been preparing questions during tutorial times. This meant they came to the stand interested and prepared, making the chats lively and focussed. We’ve left the zone open so students can continue to log in, ask questions and leave comments.

Careers ZoneThe project clearly works at some level, but we are keen to pilot it in different schools. There are things we’ll have to change for next time, such as shortening the log in process, which can take valuable time away from students who only have 30 minutes or less at the fair. We’d also like to try a new way of showing students how to use the site. We’ve written up a report about our second pilot, including some examples of conversations that happened in the chat and ideas for next time, which you can read here.

Posted on October 6, 2016 modmichaela in Evaluation, News, Project News | Comments Off on Careers Zone

Osmium Zone – I’m a Scientist, not just for scientists?

UPDATE: Read about the Institute of Physics’ research into gender differences in online engagement here.

In all the zones we run, we aim to include a group of scientists that show how diverse STEM careers can be. At the start of the last school year our long-time collaborators, the Institute of Physics, asked us to take this idea a step further in the November 2015 Osmium Zone.

A zone usually comprises five people working at the cutting edge of scientific research. This time the IOP wanted to fund a zone with four people who had studied Physics at A-level or higher, and now worked outside of academia and research, plus one physics researcher. The kicker? Ideally, those four people would also be in jobs that made use of their Physics education.

Why? Studies show that children are not aware that studying science gives you transferable skills valuable in fields outside of traditional research. Making children more aware of this fact is a key recent recommendation from the ASPIRES project into children’s aspirations.

...And reply they did!

…And reply they did!

The first unknown: Would people outside the usual research audience be interested? Answer? Yes. A call went out on twitter, and very quickly we had a range of people interested in the zone. The final selection of the Osmium Zone consisted of a communications officer for the Royal Academy of Engineering with an undergraduate degree in physics, a diplomat at the British Embassy in Tokyo working in nuclear disposal,  a biomedical engineer, and a data analyst for a solar company. The ‘token scientist’ was a  postdoctoral researcher studying lasers.

So what happened? We were confident that both competitors and children would still find the event as engaging as ever, away from the usual science focus. This is illustrated every year in the sister I’m an Engineer project that covers the diverse world of engineering. The metrics for activity in the zone point to to this holding true, showing busy live chats and especially high numbers of page views for the final two contestants, Aaron and Natalie.

As usual, the students were keen to understand the choices and motivations of the different experts, asking questions like “Why did you choose this job?”. This allowed the group to talk about how studying science had led them to where they were, and how that knowledge was valuable to them now.

By letting the children discover for themselves the specific details of each person’s job, questions about these careers naturally followed:  Do you enjoy helping and working with Japanese companies and cities?What did you take (subjects wise) to be workin with solar power?.

Furthermore, students who completed a survey before and after the event indicated a slight increase in wanting a job that uses science skills and knowledge, although the sample size is too small to draw conclusions. We will complete a proper analysis on this, and other Science Capital related outcomes, after analysing data from multiple zones and events.

Safe to say, the zone saw heated competition.

So a zone including non-scientists works. The strength of the I’m a… format is that it harnesses the power of connecting students with real people, regardless of background. What’s next? We’d like to run more zones in the future that demonstrate the diversity of science-related careers to school students, and maybe even zones completely unrelated to science. Why not I’m a Poet, Get me out of here?

For now we’re trialing a Careers Zone with alumni from past events and it’s already been fascinating seeing some of the places former researchers now work. Since November, even the token scientist in Osmium Zone has moved to a non-academic role. Natalie now works for the Met Office, coordinating efforts to maximise the impact of research into climate change, and proving further that studying science can take you to interesting places. 


Read the Osmium Zone Report for more information about the zone

Read an interview with Keith Franklin about his experience as part of the zone

Posted on September 1, 2016 modantony in Evaluation, Event News, News | Comments Off on Osmium Zone – I’m a Scientist, not just for scientists?

Demand vs. Capacity — An update for June 2016

Any avid readers of our project blog — there must be at least one of you — will have noticed we’ve written a lot recently about our increasing demand for classes, and our over-subscription rates.

In January, we published some numbers. Since then we have run the March 2016 event, and finalised class places in the June 2016 event. Here are some updated numbers:

Graph of I'm a Scientist UK class requests by event in academic year — March 2013 to June 2016

I’m a Scientist UK class requests by event in academic year — March 2013 to June 2016

In January, we wrote about the decreasing popularity of the June events, with more teachers moving to November and March.

With the spectacular growth in November 2015, we speculated that teachers may be moving from March and June to earlier in the academic year. If that was true, then we would have expected a lower demand in March and June this year.

What we see is a steady increase in demand in the June events. The growth in March 2016 was perhaps less than may have been expected given that of previous years. This could support the idea that teachers are opting to take part in November instead of March. The narrative is not entirely clear though and we need to look into this a little more.

As was true in January, what is clear is the decreasing capacity for classes in all of the events. Though this does though coincide with an increased capacity in our other projects; with the demand for classes increasing we need to increase the capacity. We need more funding.

For the past year or so we have been charging international schools wishing to take part, generating a few hundred pounds. This June we began asking the same charge of independent schools. Overall, the response has been positive — which to be honest has been a pleasant surprise. There’s a separate post to come on that, but the fact that teachers are willing to pay shows they value the activity, and that this could be a valuable funding stream in the future.

Posted on May 23, 2016 Moderator - Josh in Capacity, Evaluation, IAS Event, News, Project News | Comments Off on Demand vs. Capacity — An update for June 2016

On student activity and simplified registrations

TL;DR: We made the site easier for students to access and the proportion of our audience engaging increased.

Like any company, we have a set of Key Performance Indicators which we use to keep an eye on how well we’re doing. One of those KPIs (professionals use abbreviations) is the percentage of active students during an event, what we will call, %AS.

%AS shows the number of students who log in to the site, and go on to — at the very least — ask a question, write a line of text in a live chat, cast a vote, or leave a comment. Basically, it shows the proportion of our audience who are actively engaging with the activity.

In July 2014 Rosie posted a message on our project management app of choice, pointing out that the %AS for the previous events had been falling to the level it was at during the project pilot.

Graph of percentage of active students per IAS UK event. June 2011 to March 2016

Graph of percentage of active students per IAS UK event. June 2011 to March 2016

So, what did we do about it?

If you looked at the graph, you’ll see that we’ve already given the game away (but this is a one-graph-blog-post, and we’re not about to pad this out with multiple views of the same graph).

We started pre-registering students.

By visiting schools to observe students taking part, we saw that asking students to create their own accounts was taking way too long, was way too complicated, and largely, unnecessary.

We completely stripped down the process students go through to first get access to the site.

Previous and updated student access process

Previous and updated student access process

Previously, students would use an “access code” to get to a registration page, where they create a username and password, give us an email address, answer some other questions including some evaluation questions on their views of STEM. Now, students are given a generic username and password which gives them instant access to the site. From there, they can choose to go in and answer the evaluation questions, create a display name, and fill in their profile. But if they choose, they can get instant access to the live chats, to the question page, to scientists’ profiles.

The moral of the story then… By observing students use the site, we learnt that the registration process was too complicated. Pre-registering accounts for students does add a little more time and admin to the running of the event than not; but effort that pays off by making the site simpler to use and access for the students taking part.

Posted on May 23, 2016 Moderator - Josh in Evaluation, Event News, IAS Event, News, Project News | Comments Off on On student activity and simplified registrations

Demand for classes

Column chart showing demand for classes outstripping supply

When we started I’m a Scientist we used our March events to allow teachers to test the event and for them to come back in June with more classes. That time after school exams when teachers looked for something to inspire kids was the ideal time.

Times have changed since then. March remains popular as teachers look for activities to coincide with British Science Week, but June is no longer the most popular time of year. We think this is due to increasing numbers of schools starting the next year’s timetable and curriculum after exams and that fallow period is no longer fallow.

November is where we saw some spectacular growth in 2015. We’re not sure what has driven this – we’ll investigate and we’ll rebalance when in the year we run zones.

The other clear narrative from this graph is that excluding the lower than expected level of requests in June 2014, demand for I’m a Scientist is increasing and outstripping demand. The past four events have been oversubscribed and March looks like going the same way as November 2015. There are some advantages in terms of zones being busy, but we need a balance.

Posted on January 12, 2016 ModShane in Capacity, Evaluation | Comments Off on Demand for classes

November 2015 – Bigger than ever

We’ve just published the latest batch of zone reports for the recent November 2015 round of I’m a Scientist. You can find them here, or at the bottom of this post, but before you dive into the pretty pie charts and wicked wordles, we felt it would be useful to provide some context to keep in mind as you read:

We had an inkling that November’s eight zones were going to be big. As we reported beforehand, due to extremely high demand from teachers, it was the first time ever that we had to turn schools away from I’m a Scientist. We also had to limit the number of classes of those taking part to one or two per zone to fit as many schools in as possible. Without doing this we would have had to run double the number of zones to cover the demand.

So, after the hype, exactly how big were the zones in November? Here are some headline figures:

  • 481 students, on average, logged in to each zone. Compare this to the historical average of 353, and a target of at least 300 per zone.
  • Students submitted an average of 956 questions per zone and half of the zones saw well over 1000. The event average is 717 questions.
  • 20 live chats per zone was the average number in November. This is also usually the maximum number we allow.
The averages for the November 2015 event and the averages over three years of I'm a Scientist UK. '% of students active' is percentage of students who logged in and used ASK, CHAT or VOTE. Asterisks denote new record averages for an event.

The averages for the November 2015 event and the averages over three years of I’m a Scientist UK. ‘% of students active’ is percentage of students who logged in and used ASK, CHAT or VOTE. Asterisks denote new record averages for an event.

While the figures above are incredible in terms of numbers of students interacting with the site, high zone numbers aren’t everything. In fact, what’s best for both students and scientists is running smaller zones and more of them.

We still had great feedback from the students, teachers and scientists in November, but the bigger a zone gets, there are potential consequences. Scientists can only do so much in two weeks and with so many live chats happening, we did see slightly less scientists per chat in some zones. There’s also less chance for students to come up with their own ASK questions that haven’t been asked before, and then receive answers made just for them. The lower proportion of approved questions to submitted questions reflects this.

We don’t want these consequences. We want every student to have their own high quality, personalised interaction with a scientist. An increased number of smaller zones would make this easier for everyone.

The situation we’re in also means we have to choose which schools take part, and in an effort to reach those who lack regular STEM engagement, we’ve started prioritising places for Widening Participation and rural schools. Ideally, however, we want all schools who register for an event to get a place in a zone.

Thanks to our current funders, and the sterling efforts of the scientists, November was a success. But if we’re near the limit of zone size now, what’s next? Luckily, the high demand for I’m a Scientist represents a huge opportunity for organisations to reach the public. Being oversubscribed means we can show there are hundreds of schools ready to engage with hundreds of keen scientists. All that’s needed is the funding for the zones to let them get on with it, and we’re going to try our best to make that happen.

November 2015 Reports:

Posted on December 17, 2015 modantony in Capacity, Evaluation, Event News, News | Comments Off on November 2015 – Bigger than ever

3 words to describe a scientist

Survey results from a Year 6 student - Swanmead Community School

Survey results from a Year 6 student – Swanmead Community School

In March 2014 we started running zones for primary students only, increasing the number of primary school students who take part in I’m a Scientist. And we wanted to know more about how the event affects them.

This time, we drifted away from our usual online environment and sent printed surveys to teachers taking part in the Colour Zone and the Thulium Zone, the two primary only zones we ran in June 2015.

We also wanted to check if the response rate to printed surveys was higher than the response rate to digital ones, and it is. We got pre and post event results from roughly 40% of the students who took part in the two primary zones. Usually around 10% of students who take part in the events fill in both the pre and post event digital surveys. Analysis and interpretation of online surveys is quicker and more efficient that printed ones, but getting teachers involved in the evaluation process definitely made a difference.

We asked students which 3 words they would use to describe a scientist, before and after taking part. We got responses from over 300 students from 9 different schools.

  • The students used more that 230 different words to describe scientists. They used 188 different words before taking part and 127 after, maybe indicating that they are clearer about what a scientist is after taking part. Awesome, cooperative, inquisitive, life-saving… are just some examples of the words they used.
  • ‘Intelligent’, ‘clever’ and ‘smart’ were the most popular describers, although students used them less often after taking part – 46% of students used at least one of these words pre-event compared with 40% post-event.
  • Students described scientists as ‘awesome’, ‘epic’, and ‘cool’ twice as many times after taking part in I’m a Scientist– 9.7% of students used these words post-event compared with 4.4% pre-event.

We grouped words in synonym clusters, and we represented the relative pre to post-event difference on word use by gender:

new graphOnly words that were used at least 10 times are represented.

From the graph you can see that girls and boys equally use less negative stereotype words like ‘crazy’ after taking part.  Emotional descriptors – like ‘fun’ or ‘exciting’ – are mentioned more frequently after the event. You can also see that boys are more likely to use the word ‘fun’ after the event than girls. However, girls were more likely to use the word ‘awesome’ after taking part.

What do you think? Is there anything that particularly calls your attention? We asked a primary schools teacher and she told us:

I’m not surprised by the ‘awesome’ leap – I don’t think children really understand what is involved in the daily work of scientists until they interact on I’m a Scientist. My own students have been blown away by talk of live sheep spines, looking at volcanoes in space, etc.” – Tracy Tyrrell

If you are a primary school teacher and you want to evaluate your students’ attitudes to science before and after doing I’m a Scientist, you can download PDF version of them here:

If you are an academic (or, really anyone!) and you are keen to see and analyse the raw data of these printed surveys, please let us know in the comments and we’ll share them with you.

Please leave a comment, let’s continue the dialogue.

Special thanks to Dr Jessica Hamer, for her advice on the the printed surveys design.

Posted on November 2, 2015 modangela in Evaluation | Comments Off on 3 words to describe a scientist