We often hear directly from scientists about the positive impacts they get from engaging with our enthusiastic and diverse audience of school students. We’ve been able to research these impacts in more depths as part of the ChallengeCPD@Bath project run by the University of Bath. This in turn is part of the Strategic Support to Expedite Embedding Public Engagement with Research (SEE-PER) project funded by UK Research and Innovation.

We looked at whether taking part helped scientists from the March 2018 event improve their communication skills and if so, explore why IAS worked as experiential training. The results showed scientists became more confident communicating with public audiences and helped them develop effective ways of talking about their work. The opportunity for frequent practise, honest student feedback, and the text based nature of interaction were among the elements highlighted by scientists as particularly helpful.

Using these results we can demonstrate I’m a Scientist is effective experiential training in public engagement, especially for scientists with less experience or lower confidence. We’ve already incorporated some of the evidence into new materials explaining the value in taking part. The strongest evidence currently comes from the qualitative research and we can look at ways of more effectively assessing the positive impacts through quantitative data.

Read more about the results and methods for the elements of the research:

1. Scoring confidence in communicating
2. Rating scientists’ sentences
3. Follow-up interviews
4. Methods

We’d like to thank Dr Helen Featherstone and the team at University of Bath Public Engagement Unit for the opportunity to be part of the ChallengeCPD@Bath project. Find more information on the other outcomes of the project at the Public Engagement Unit blog.

Taking part improves the confidence of scientists communicating their work to public audiences

Anecdotally, we often hear that taking part gives scientists a lot of confidence when talking about their work. We decided to 1.Ask scientists about their confidence before and after the event, and 2. See if tracking how they felt each day might reveal more evidence of experiential learning.

We asked scientists to score their confidence communicating with the public each day of the event. We hypothesised that once scientists started the event and had a go at actually talking to the students their confidence would dip, before increasing again as they became better at handling fast paced live chats and imaginative ASK questions.

Out of a possible 517 scores, the scientists submitted 414, with scores from 37 scientists submitted each day on average.

The distribution of final scores submitted by scientists compared to their earliest score, either from before, or in the first week of, the event.

Overall, there was a clear improvement in the scores submitted after the event. For example, 91% of scientists scored themselves at 4 or higher post-event and, in particular, 44% scored themselves at 5 (very confident) post-event, compared to just 13% beforehand.

Day to day, there wasn’t a dramatic dip then increase. However when split into two different segments, a clearer pattern could be discerned:

1. Those less confident (3 or less) at the start grew the most as the event went on.
2. Those more confident (4 or more) at the start became less so once the event started, before becoming more confident again in the following days.

Judging by the comments submitted with scores, it could be that Group 1 contained people who had done little public engagement beforehand, whereas Group 2 could include those who were more experienced.

Both groups finished with almost exactly the same score, suggesting the experience was beneficial no matter where you started. A couple of scientists also noted that being evicted had decreased their confidence slightly.

The scores also suggest women gain confidence at a faster rate than men. The confidence of female scientists increased on average by 27% during the event (3.52→4.45), compared to an increase of 10% for male scientists (3.76→4.13).

2. Quantifying skills improvement using one sentence descriptions is difficult

Every scientist who applies for IAS must submit a sentence that describes what they do. These are then judged by students and teachers and the average ratings are used to help guide who gets places in each event. We wanted to see if the ratings increased for the sentences written post-event, and if their readability had increased indicating that the scientists were better at communicating what they do. 

35 scientists submitted a post-event sentence to compare with their pre-event sentences.

Student/teacher ratings: The average for post-event sentences was 0.77, compared to a pre-event average of 0.90, representing a decrease in rating of 0.13

Readability: The average grade for post-event sentences was 9.23, compared to a pre-event grade of 10.32, representing an increase in readability of 1.09 grades.

The results do not suggest any clear conclusions. The decrease in student ratings, although indicating that the sentences were likely not as good, is very small. Likewise, the increase in readability is too small to infer impact. Subjectively, there is a sense that less exciting language has been chosen in the post-event sentences in favour of conciseness.

This could indicate either:

1. Scientists do not become better communicators through the event
   • This is countered by hundreds of scientists reporting increases in their communication skills and continued use of effective phrases developed in the event.
2. Other factor(s) influenced the test, for example, the motivation to impress the students was missing after the event had taken place.
   • If run again, we could provide a reward for the best rated post-event sentences to match the pre-event motivational condition. However, it may be hard to replicate the incentive provided by the opportunity to take part in IAS.

3. The format enables and encourages scientists to develop their communication skills

Hearing from the scientists interviewed, it was clear that all five believed taking part in I’m a Scientist had improved their skill in communicating with the public. Read the full narrative for each interview. 

In summary, benefits mentioned were:

• Being able to better adapt their language to their audience (avoid jargon and technical language)
• Developing succinct phrases and effective analogies to explain concepts that they continue to use
• Getting better at explaining the wider relevance of their research to get people engaged, even if they had had lots of previous outreach experience.
• Positive impacts on how they communicated in professional contexts.
• More confidence when communicating, especially for those with less outreach experience. All had taken part in further outreach activities following the event in March.

The format of I’m a Scientist was particularly helpful for the scientists in the following ways:

• Frequent practice – Daily chats and ASK questions over two weeks allowed for new ways of explaining to be tested out and phrases to be honed over time
• ‘Two-way’ – Student feedback in the form of further questions let scientists gauge how well they were communicating
• Online and text-based –  Students gave more honest responses than some scientists had experienced in person, scientists felt more comfortable answering challenging questions.
• Shared experience –  Scientists could see and learn from how others in their zone approached answering the students

“…every time I was answering to the kids they would ask more and more and like that would sort of bring, break down the concept into basics and I understood that that is how you can explain your work to a non-specialist audience” Kezia, PhD Student, no previous outreach experience

“Some of the analogies I totally plagiarised from the students! They come up with their own ways of understanding what you do. You’d describe it and they’d say ‘oh, is that a bit like this?’ and you’d say ‘exactly’. And that’s really nice too, when they do the imaginative work of explaining it for you. I really liked that.” Max, PhD Student, some outreach experience.

“I think I’ve learned how to be more ‘less specific’, how to maybe explain more of the sort of bigger picture of where my research could go and then discuss into more details from there depending on if there’s interest. I think that helps people who don’t understand in technical terms more what I’m more what you’re doing if you sort of say well this is the end goal” Neil, Postdoctoral researcher, experienced in outreach

“It put in my mind that I can be asked any question outside my field that I don’t expect… I would do my best to answer those questions. And if I don’t really know something, I wouldn’t be afraid. I would say ‘Ah I’m sorry I don’t really know about this’ And I could say ‘What do you know about this?’” Walaa, PhD student, some outreach experience

“Now I feel I can, you know I’ve got that you’re in the lift and they’re getting out on the second floor speech….Because you’re not in person… there’s not that level of politeness, it’s just ‘I’m not interested anymore, bye, I’m going to speak to someone else’, rather than feeling like they’re stood in front of their teacher, or their mum and dad and feeling like they have to be polite to this person they are talking to…  I kind of think it makes you have to get better.” Aileen, PhD Student, experienced in outreach

 

4. Methods

Scoring confidence in communicating

A short survey was sent to every scientist on the Friday before the event asking them to score themselves from 1(Not very confident) to 5 (Very confident) in answer to the question ‘How confident are you feeling at this moment about communicating your work to a lay audience?’. There was also an open text field to record their thoughts on how they had arrived at that score.

The scientists then received the same question every weekday at 4pm during the event until the final Friday. This gave us a maximum of 11 responses for each scientist. Reminders were also sent first thing each morning in case they had missed the previous day’s email.

Rating scientists’ sentences

Students are asked ‘Should this person get a place in the next event?’ and then rates each sentence either Definitely not, Maybe not, Probably, Definitely! We then assign these options a rating from -2 to +2 to give an average rating for each sentence.

In March, we asked the participating scientists to write a new sentence describing their work after taking part and compared its average rating to that of the sentence they submitted in their application. Each sentence was rated an average of 18 times. We also processed the sentences individually through an online readability scorer using the Flesch-Kincaid test. In this test, the lower the score, or ‘grade level’ the more readable the text is.

Interviews

We carried out recorded phone interviews with a targeted sample of five of the scientists. We biased towards PhD students who have more to gain from training, and included people with varying levels of PE experience.

Here are narratives for each of the five interviews carried out as part of the I’m a Scientist element of the ChallengeCPD@Bath project, funded by UKRI. These interviews investigated how I’m a Scientist works as experiential training for scientists communicating their research. Read about the full report post on this research

Kezia – 1st year PhD student

I’m a Scientist was Kezia’s first outreach activity and she was certain it helped her become better at communicating her work. She says the experience gave her the confidence to do more communication with public audiences and she has started delivering workshop sessions related to her research for school students. Kezia also noted she has learned to adapt her language when talking to ‘non-specialists’. In particular, she now uses analogies to explain concepts related to her work, and says this approach came from her I’m a Scientist experience.

“It sort of made me realise that yeah, I am confident, yeah, I can do that (outreach activity)… I’ve also signed up for the same sessions to be done next year with the same schools.”

“It improved how to explain my work to the common audience so I’ve been able to talk about my work with even my non-scientist friends… I’ve been able to think of and come up with some sort of analogies when I’m explaining concepts to people, like I’ve never thought of that aspect before I took part in the I’m a Scientist contest”

What was it about IAS?

The iterative nature of I’m a Scientist helped Kezia adapt her language and develop analogies. For example, she says that the multiple follow up questions from students in response to her answers in live chats forced her to find more and more accessible language and new ways of explaining her work.

“...every time I was answering to the kids they would ask more and more and like that would sort of bring, break down the concept into basics and I understood that that is how you can explain your work to a non-specialist audience”

Any changes in a work context?

Kezia not only feels she has become a better communicator in a professional context, but the improvement has been noticed by her colleagues.

“ I work in an interdisciplinary project so not everyone understands the technicalities of what I do so yeah it has actually had an impact on the way I communicate overall… I’ve got very good feedback from people unlike before so there has been some sort of influence.”

Anything else that stood out?

Having done no outreach previously, Kezia found the online format appealing as a ‘first step’.

“I’m from India so I hadn’t done any outreach back there so it’s just not the research culture over there, so it was a completely new thing to me so the IAS forum was an online forum where I felt a bit safe and confident”

Aileen – 1st year PhD student

Before I’m a Scientist, Aileen had had regular experience communicating her work to visitors at her research institute and at one off events like Big Bang Fair. She believes taking part helped her develop short ‘quick-fire’ explanations of her research, improved how she adapts explanations for non-scientists of all ages, and improved her ability to deal with unexpected questions

“…you know when somebody wants a ten second answer  of what do you do? Rather than going in to the whole ‘I do this this and this and I’m a PhD student blah blah blah it’s very easy when you do it all day to give them five minutes and not even scratch the surface whereas now i feel i can, you know i’ve got that you’re in the lift and they’re getting out on the second floor speech.”

What was it about IAS?

The need to regularly get across information ‘succinctly’ in live chats particularly helped Aileen develop her short explanations. She felt that because students could easily switch their interest to someone else, without worrying about being polite, she was forced to explain herself as clearly and quickly as possible. Aileen also appreciated the information about the class’s age or level before the chats to help her adjust to what they might know about already. The computer based, text only nature of the activity also meant she was comfortable taking a moment to compose answers to unexpected questions

“…Because you’re not in person… there’s not that level of politeness, it’s just ‘I’m not interested anymore bye, I’m going to speak to someone else’, rather than feeling like they’re stood in front of their teacher, or their mum and dad and feeling like they have to be polite to this person they are talking to…  I kind of think it makes you have to get better.”

“…It’s you know preparing you for the unexpected, it’s quite good and then you’re behind the screen so if you suddenly have 30 seconds when you clam up it’s a lot easier than when you’re standing in front of someone. So it’s quite good to practise it in that format, rather than ‘oh god there’s a major investor stood in front of me and I really need to answer them now.”

Plans for, or effects on, further public engagement

Brilliant Club – Aileen is looking to being more targeted with her outreach as she goes into the second year of her PhD (so an effect not related to her IAS experience) and preferring something where she can build relationships over a longer period of time.

Any changes in a work context?

Aileen believes her elevator pitch has helped her in professional contexts, such as conferences.

“in terms of turning up to conferences and that kind of thing, being able to explain what I do in a very short speech is helpful.”

Anything else that stood out?

Convenience of IAS a big plus for keeping her supervisors happy, perhaps pointing to the perception of outreach as ‘not real science’

“…I can do it over lunch  and I can still do, you know, maybe not the same amount of work that I would do on a normal day but I could still do 80% of the work that I could do normally …. and my supervisors don’t get upset about doing too much time doing outreach and not enough time doing my actual experiments.”

Max – 3rd year PhD student

Max is a 3rd Year PhD student who has previously written blogs and taught science at summer camps and in Oxfordshire. This outreach has not involved much of his own research and “that definitely sets I’m a Scientist aside, where the main focus was me and my work.”

By taking part Max says he has now developed a ‘better arsenal of metaphors and analogies’ for describing what he does, and feels he has a much better understanding of what public audiences find interesting about his work. He also feels more confident in talking about his work since taking part.

“…often you hear scientists being afraid of boring people or not being interesting, so you err on the side of not wanting to talk about your science at all… The experience showed me that actually, in general, people are interested in hearing about what you do, you just need to find the right way to engage their interest. And probably the way that you do that for a 14 year old is not that different to the way you do it for a 40 year old, especially if their knowledge of science is kind of comparable anyway.”

What was it about IAS?

Max thinks that the ‘two-way’ nature of I’m a Scientist was key to the impacts on his communication skills. Responses and feedback  from students directly helped him develop new analogies, and told him when he wasn’t explaining himself well enough.

“Some of the analogies I totally plagiarised from the students! They come up with their own ways of understanding what you do. You’d describe it and they’d say ‘oh, is that a bit like this?’ and you’d say ‘exactly’. And that’s really nice too, when they do the imaginative work of explaining it for you. I really liked that.”

“The whole thing was very two-way — you’re giving answers to the questions people are directly asking you, and then you get their responses as to whether or not they like the answers, both immediately because they’d say ‘that makes sense’ or they’d ask another question, and also long term because of the votes and the competition”

Plans for, or effects on, further public engagement

Max won his zone and plans to use his prize money to develop a podcast about his field. It will feature local school students asking questions about gene editing, and agrees that how I’m a Scientist lets the students lead the discussion while providing stimulus material (profiles etc) has influenced his approach.

“To me, that kind of two-way engagement where you’re actually having a scientific discussion, pitched at the right level, is just so much more productive than Brian Cox asking you to think about the universe or whatever.”

Any changes in a work context?

Max feels that answering the students’ ‘inventive’ questions has led to him talking much more with colleagues in the lab about their work and his own, and in a way that is more meaningful than technical.

“I think the amount that I chatted with my colleagues when I had these crazy inventive questions from students in I’m a Scientist, made me see that if I had that same level of discussion with them about my work — not just ‘how do I attach X to Y in this machine?’ but also ‘what does this actually mean?’ — it could be really productive. I think in general I’ve been talking to my colleagues more and been trying to see the wood for the trees a bit more and actually talk about the concepts, rather than just the nitty gritty, which as technical scientists we always get bogged down in.”

Anything else of note?

Max’s university, and others, approached him to write blogs posts about taking part in IAS.

Neil- Postdoctoral researcher

Neil does regular chemistry outreach as part of a tour of Scottish schools twice year, however it doesn’t involve talking about his own work. He says he saw a big change in how he communicated in both ASK and chat over the course of the event. He thinks taking part has improved his use of appropriate language and has changed how starts conversations with public audiences, now focusing first on the ‘bigger picture’ and the relevance of what he does to other people.

“I thought I was relatively good at communicating, obviously I still think I do, but I think my levels  of technical, what my perceived level of technical detail that people will understand and actual level may be a little bit different.”

“…now I know, I feel I know the level that I should be trying at and the level that you get more response from.”

“I think I’ve learned how to be more, more ‘less specific’, how to maybe explain more of the sort of bigger picture of where my research could go and then discuss into more details from there depending on if there’s interest. I think that helps people who don’t understand in technical terms more what I’m more what you’re doing if you sort of say well this is the end goal, rather than just saying we do this because it’s interesting or we do this because it’s an important piece of fundamental research that will lead on to more things. You sort of need to explain what the other things are and then come back abit to say why what you’;re doing fits into that”

What was it about IAS?

The back and forth conversation with students (or lack of, depending on his answers) helped Neil adjust the language he was using and the way he was explaining things, and the text only format was important for this to happen. He also learnt a lot by seeing how another scientist with lots of outreach experience was approaching their answers during live chats.

“If they’re asking me about my work or what I do day to day, erm, I think when I started answering them I think I was trying I tried to be a bit sort of over technical and not simplify enough and  I could see quite quickly that ‘hmm that’s not quite right so, cos you could see in the chats maybe you didn’t quite get more response’

“I definitely learned quite a bit from especially from Lauren Webster, who’s a chemist from Dundee, cos she also does a lot of outreach at the moment. She used simpler language from the start … I found that if I sort of changed how i was answering things, maybe made things a little more vague, but more understandable, the conversations went on for longer”

“…text based, that’s how they’re used to communicating, it’s less formal and if you’re dealing with a number of people I’m guessing it would be quite hard to have a two way video conversation”

Plans for, or effects on, further public engagement

Neil thinks there was far more ‘interactivity’ with students in IAS compared to his schools workshops.

“…as soon as you’ve delivered the activity or the demonstration, you’re tidying up to do the next one and you’re thinking you know ‘right , i’ve got to go to the next place and get sorted again.’ This, there’s a lot more interactivity here and that was good.”

Anything else of note?

Neil enjoyed answering the science questions more than ones about himself outside of his work. He also found the chats with fewer scientists more enjoyable because it focused the conversation.

“…when there was only two or three of us (scientists) it was much easier to have thready conversations where both of us answering to the pupils and building on each others’ answers so I think we got more out of it when it wasn’t quite so chaotic”

Walaa – 1st Year PhD Student

Walaa had done a short talk competition last year and made the regional finals. After taking part in IAS she felt she had increased her confidence when dealing with unexpected questions, and learned how to provide more information for interested people. Walaa also feels she has gained a better understanding of what the public thinks and knows about the area of her research, bees.

It put in my mind that I can be asked any question outside my field that I don’t expect… I would do my best to answer those questions. And I would say, if i don’t really know something I wouldn’t be afraid. I would say ‘Ah I’m sorry I don’t really know about this’ And I could say ‘What do you know about this?’

What was it about IAS?

Walaa found the event was a good opportunity to practise explaining herself without technical language.

“I tried to start from the beginning, with very basic things, easy things. They don’t know any technical language so i tried to avoid using technical language, or if I have to use one then i would explain it, what does it mean , break it down into parts”

Plans for, or effects on, further public engagement

Walaa has since signed up as a STEM ambassador and delivered two workshops about bees

Any changes in a work context?

Doing IAS has been part of a process for Walaa about learning to sell herself

Anything else that stood out?

She feels the having IAS on her CV got her a job translating science writing into Arabic.

It’s well established that scientists doing I’m a Scientist gain just as many positive outcomes as the students they talk to. Using the Vitae Researcher Development Framework to frame these benefits allows us to articulate them in a way that resonates with universities.

Recently, we surveyed event alumni from centres of doctoral training (CDTs) to quantify the effect of taking part on relevant RDF descriptors. This is part of our ongoing strategy to build the evidence base for IAS as a CPD activity which departments can offer to their students. In total, 37 alumni who had taken part in I’m a Scientist or I’m an Engineer at least 3 months prior responded.

Pleasingly, we saw positive effects across the board. Here we breakdown the key results by relevant RDF domain (don’t worry, we weren’t interested in all 60+ descriptors).

RDF Domain B – Personal effectiveness

81% of respondents considered taking part as contributing to their Continuing Professional Development (B3 Continuing professional development B3)

73% say that taking part increased their enthusiasm and passion for their research (B1 Enthusiasm, B1 Self-confidence, B1 Self-reflection)

• Getting to reflect on their personal motivations and the interest that students showed in their research were the biggest contributing factor to this effect.

68% had seen benefits to either their professional networks, the number of opportunities offered to them, and their profile and reputation as a researcher. (B3 Continuing professional development B3, Responsiveness to opportunities, B3  Networking )

“It seems to have been a big positive on my CV, which I have to admit I didn’t expect. I was asked about it specifically both at interview, and later by my line manager during the induction process.” – CDT student

“It establish me as an enthusiastic researcher willing to engage with researchers. if future employers google me they find something positive straight away.” – CDT student

RDF Domain D –  Engagement, influence and impact

94% said taking part improved their ability to communicate research with public audiences

• Being able to ‘adapt language for different ability levels’ was the most improved skill among respondents

92% said they adapted their approach to communicating with students as the event went on

91% continue to use phrases they developed during the event to explain what they do in other contexts (D2 Communication methods, D2 Communication media, D3 Public engagement, also A3 Argument construction)

69% said taking part had improved their understanding of the impact of their research on society in some way (D3 Society and culture, D3 Global citizenship, D3 Public engagement, also C1 Ethics, principles and sustainability)

• The impact of their research on people’s everyday lives was the area most had improved this understanding. Understanding of ethical issues and the economic value of their research were other improved areas.

“Writing an answer, it’s much easier to stop using jargon, it was a good lesson in recalibrating yourself… Now when you go to do face to face outreach you feel more equipped.” – CDT student

‘”Engineers typically do not have a lot of time to give back to the society in the form of public discussion, but this experience highlighted that it is very important to have those conversations with new generations coming through and as a result I have participated in more public engagement activities than I previously would have.” – CDT student