We know anecdotally that participating in the I’m a Scientist event has a positive effect on students’ attitudes to science. Teachers tell us that their students were buzzing, and that they understood more about science. Students thank scientists in live chats for an interesting lesson. They tell us that it is “better than Facebook”.
But being data geeks that isn’t good enough. We wanted to know just how much we were affecting the students. Were we affecting them all by the same amount? Girls, boys, year groups? Did being more active in the event mean a greater change in attitude?
Questions we ask students
Thanks in part to the pilot work by Robin Longdin, then a SciComm masters student at UWE, we ask every student registering for I’m a Scientist a set of 4 questions:
- How does school make you feel about science?
- Are you planning to choose a science subject at the next stage of your education?
- Do you think jobs involving science are interesting?
- When you finish your education, how likely are you to look for a job that uses your science knowledge and skills?
We also ask the same questions towards the end of the event and ask for the username so we can match against their initial answers, their gender, year group, zone and activity levels within the event.
Measuring attitudinal change
The important thing for us is not the answers they give but the difference between the beginning and end of the event. We want to know if after two weeks of being exposed to our scientists they feel differently.
Each question had a five point answer scale. We intentionally wanted a neutral middle answer. We can’t expect students to all feel passionately about science. We then allocated a numerical value to each answer. 2 for the most positive, -2 for the most negative. We then simply subtract the starting answer value from the end of event value to give us a value for change in attitude for each of the four questions for each of the student who answered.
Do more active students see a greater change in attitude?
There is a lot of data generated and one of the challenges is to consolidate it into meaningful clusters. For example what do we mean by levels of activity? Some students are lively in live chats. Others ask a lot of questions. Some do both. We looked at the 3 main types of activity: ASK, CHAT, VOTE and scored each student between 0 and 3 depending on how active they were. We then totalled the individual scores to give each student a score between 0 and 9 overall. That is the X-axis below.
The first thing to note is that the overall trend is that the more activity on the site the more positive the change in attitude. Secondly the change in attitude to the final question: “When you finish your education, how likely are you to look for a job that uses your science knowledge and skills?” is much lower than the other questions, and in fact sometimes negative even for those students who made the most of our erstwhile role models.
We need to investigate this further. It could be a flawed question? What does it really mean to 13 and 14 year olds. Do school students consider a PhD studentship to be a job? Are half our role models not helping with that question?
We went back to the source data and found to our embarrassment that we had made a mistake and we were comparing the answers to the final question to the answers to the 3rd question. D’Oh! The good news for us is that when you use the right data, the story looks even better. Students emerge from I’m a Scientist feeling much more positive about jobs in science for themselves.
The greatest effect is on girls and Year 10 & 11 students
This graph shows the average change in attitude for 6 groups by question asked. It clearly shows that the event has the greatest effect on girls and on students in Years 10 & 11.
This is based on a total of 853 valid responses from students across 3 events from June 2013 to March 2014 in the UK. The n for the groups in the second chart are:
Years 7-9: n=589
Year 10 & 11: n=130
They don’t all add up to 853. Sometimes it is a group of girls and boys who register. We also had 21 6th formers respond, but that is too small a sample to use. And sometimes the default “Please choose” answer remained. Yes there are disproprtionately more girls than boys, but yes, more girls than boys do participate.
Leave a comment and start a conversation!
Finally, we do this analysis for our own sake so that we can better understand our event, but we publish it for your sake, whoever you are. We want other practitioners to share in our knowledge. Please leave a comment to say “thanks”, or to ask for clarification or for more information.