Research vs Evaluation

Science Learning+ is a significant funding scheme provided jointly between the Wellcome Trust and the National Science Foundation.

Learning can happen anywhere and at any time. Science Learning+ is an international initiative that aims to understand the power of informal learning experiences inside and outside of school.

The second aim of the scheme is to

“bridge the practice and research gap”

At a seminar in July aimed at providing an update on the Phase I project an interesting conversation developed about that gap between Science Communication practitioners and researchers.

I heard one speaker talk about practitioners wanting to know if a hypothetical red headline would give a 3% uplift in visitors. I responded on twitter:

Not all practioners agreed with me. Some felt each project would be unique enough to warrent a rewriting of expectations

Others simply disagreed and place efficacy as something for researchers:

In the end 140 characters felt underpowered.

For me research and evaluation are different, but very related.

I expect research to tell me if an approach to science communication works and how it works. I expect evaluation to tell me how well a project is working and how it can be improved. I would like evaluation to draw upon the research to extrapolate that particular  activities will lead to particular outcomes.

For example using I’m a Scientist:

The feedback we get from our participants is that connecting online with scientists improves their attitude to science, and to jobs in science. We seem to find the changes in attitude among girls is greater than it is for boys.

I want some research to tell me why those conversations are improving attitudes and if those changes are persistent. I want the research to be telling me how online activity compares to offline activity and why.

I want research to tell me what characteristics of engagement deliver the best and most persistent improvements in attitude and achievement.

Then I want my evaluation to examine our work against those characteristics and to suggest ways to improve them.

Research  = why something works

Evaluation = how well something works

What do you think?

Posted on August 27, 2015 ModShane in Evaluation, Science Engagement | 1 Comment

Harwell Open Day Zone Report

HRWL LEGO PosterOn Saturday July 11th the Harwell Research Campus opened its doors to the public. 10,000 people were expected to come and see some of the most complicated and cutting edge science equipment in the world. Hundreds of volunteers working with that equipment were on hand to speak with the public and show them what happens.

We created a Harwell Zone to allow visitors to text in questions. It was promoted using posters as above.

The logic was that with 10,000 visitors some would not get to ask their questions, or may think of it later, or possibly be too shy to ask in person. It was a family day.

Working with the over-worked (understatement klaxon just sounded) outreach team at Diamond and Harwell we set the zone up and promoted it to exhibitors.

It wasn’t a great success. A mere 14 questions were sent.

However in the interest of continual improvement let’s look at what worked, what didn’t and most importantly what would be do differently if given the chance.

What worked?

The technology did. Visitors were able to text a question, get a response to manage their expectations and a notification when someone answered the question. We also had an enthusiastic response from exhibitors who signed up. However the enthusiasm was far far greater amongst I’m a Scientist alumni. It seems that we didn’t really reach many new scientists. It was in the main scientists we already knew who happened to be already exhibiting.

What didn’t work?

We didn’t reach new scientists. The vast majority of scientists there were not aware that they could have taken part. This meant that the online offering was very patchy. As you wandered around the trained eye (mine) occasionally picked out a poster. The only exception was the RAL Cyrogenics Lab where Vicky Bayliss had printed out extra posters and placed them all around the lab.

We didn’t reach the public. There simply was not enough publicity.

What would we do next time?

  • An online Q&A offering needs to be an integral and supplementary part of the open day. Exhibitors should be opting out not in. Taking some questions online should be seen as part of the overall experience. We need to brief potential participants better.
  • The online experience needs to match the offline experience. At Harwell some participants were running a desk, others signed up the entire lab. It was disjointed. It would work at the lab level better.
  • We should pay more attention to the offline visitor flow. At Harwell many labs had a distinct flow from start to finish. We should have been making sure as people left they not only handed back their lanyards but took away a leaflet offering the chance to ask follow-up questions.
  • Use the online zone to supplement the offline experience. At Harwell visitors were invited to view the Diamond Light Synchotron but there were explainers in there. It was a prime spot to publicise the zone.
  • Programme notes. People tend to keep the programme notes with them throughout the day and on the way home. That’s the time to mop up any unanswered questions.
  • Make more of the online zone. We could take feedback, promote new open days, communicate campaigns, point to more resources. The point is that if someone asks a question online they are in effect asking to engage in conversation. Same as offline. The difference being that online you have the ability to invite the visitor to restart that conversation at any time in the future.

 

Posted on August 24, 2015 ModShane in Evaluation, News | Leave a comment

“I’m a Scientist is definitely the most fun I’ve had in my lab coat”

After every event we ask the zone winners to write a short blog post to be sent to all the students in who took part in the zone. It’s a great way for the scientists to reflect on the previous two weeks and thank all the students for voting for them.

Let’s take a look at what they had to say…


Jess, Colour Zone

I’m a Scientist is definitely the most fun I’ve had in my lab coat.

The questions were incredible, totally unpredictable and the scope was well beyond that going on academic circles; by the final I’d tried to prepare a few ‘go-to’ answers, “What is colour?”, “What happened during the big bang?”, “Will we ever cure cancer?..but I couldn’t predict a single one! After the 30 minute live chats my fingers were tingling so much I had to delay measuring polymers in the clean room.

I cannot express in words how much I’d recommend this to any PhD student, science technician or any academic with time on their hands.

It’s made me question every part of my research, assess how realistic I am and be more creative in my experimental design. It’s made me so proud to be a physicist, work at Imperial College and be able to help these curious kids.

James, Electromagnetic Zone

Things got serious, silly, deep, dark, hilarious, and on occasions weird; all in a 30 minute burst of direct exchange between today’s researchers and tomorrow’s minds.

It was *energising* – I loved it. And I learned a tonne of stuff myself.

Sarah, Extreme Force Zone

I really hope (and believe) more of you became enthused by science by the event, and can maybe envisage yourselves on the other side of the chat telling students in the future all about the great work you do.

Andy, Health Zone

It was exciting, loads of fun, I learnt loads of stuff I never expected to and it completely took over my whole life for two weeks! At midnight on the Saturday half way through the competition I was discussing “what is time?” with some friends at a wedding. It really made me think about things that I’d not considered before.

Laura, Hormones Zone

An event like this is a fantastic way of giving school pupils an idea of how many areas in science there are and to let them see that scientists really are quite normal people (mostly).

It was a manic two weeks but it was so much fun and VERY addictive! The Live Chats were crazy at times but always great fun and certainly kept me on my toes. Any chance I got, I was answering the question on ASK (all 214 of them!) – on my journey to work, breaks, lunch, journey home, on the way to meet friends – any chance I got!

Richard, Pharmacology Zone

Wow! What an amazing two weeks. Without a doubt, I’m A Scientist, Get me out of here has been one of the most enjoyable but exhausting experiences of my career.

The live chat sessions are insane!  Questions ranged from everyday trivia like “Batman or Superman?” through philosophy and ethics “should doctors be allowed to assist in suicides” to hard-core pharmacology “how are drugs developed?” Now…. imagine trying to handle all three of those questions simultaneously, and fast enough so that your competitors don’t get in there first with a killer answer, and you’ll have some idea what it was like and why it was such fun.

Thomas, Polymers Zone

[The students’ questions] got me thinking on all fronts – from how to describe my research, and why I actually carry it out, to appreciating why the sea is blue and coming up with a “hand-shaking and waving” analogy for pi and sigma bonding during a quiet evening – definitely a blast all in all.

Hayley, Ytterbium Zone

I was continually impressed with the questions posed by the students in both the live chats and also via ASK. My very first question to answer was about black holes (a topic waaaay out of my comfort zone!), so I’ve been learning myself from day one!

… I have learnt so many diverse facts from this event, including how chameleons are able to change colour and learning that plants may, in fact, have feelings!

Ryan, Thulium Zone

The two weeks of answering questions from you all was crazy. I’ve never gone out of my way so much to be on time for everything. I had alarms set so I could attend chats, I spent my evenings answering all the ASK questions which were left for me and others.

I had questions ranging from “how many stars exist” to “why do I do science” each question was fantastic and there were so many I actually had to stop, think about and wonder how it actually worked.

Normally I’m one quick fire answers back but there were so many questions where I had to stop and think to even form an idea of an answer.

The whole event was an experience and I recommend it to anyone to try it, you’ll learn so much not only about your own area of expertise but also about others.

Jonny, Lutetium Zone

Initially I didn’t know whether I should enter I’m a Scientist, Get me out of here. I’ve not really done much written science communication before and I didn’t know how I would come across without actually being present.

One thing I wasn’t prepared for was the breadth of the questions all of us scientists were asked. It made me realise how much I didn’t know.

Chris, Hafnium Zone

The last two weeks have been an absolute blast. The range of questions you get on the board is insane! You guys ask better/harder/faster/stronger questions  that I ever imagined coming into the contest.


If you think you can take the better/harder/faster/stronger questions… Want to discover new ways to communicate your work… Or want to learn new facts about chameleons…

Apply now to take part in the next event

Click to apply for I'm a Scientist

I’m a Scientist, Get me out of here runs every March, June, and November. For more information on the zones running in November 2015, take a look at this post.

Posted on July 27, 2015 Moderator - Josh in Event News, IAS Event, News | Leave a comment

Launching questions into the Thermosphere

From the effects of solar radiation on people living on the International Space Station, to opinions on space travel; there are few topics which inspire quite so many questions as Space.

Another question, asked probably in every live chat in every Space Zone we’ve run: “Have you ever been to space?” The answer has always been “no”… Until now.

This year, Tim Peake will become the first British ESA astronaut to live and work on the International Space Station (ISS) and thanks to funding from the UK Space Agency and the support of the European Space Agency, our latest project will bring the buzz and excitement of Tim’s mission into classrooms all over the UK.

Our latest project gives students the opportunity to put their questions to the people behind Tim’s mission — members of mission control, researchers with experiments running on the ISS, trainers, and scientists and engineers involved in the design of the ISS and its equipment — before four winning schools get the chance to take part in a live chat with Tim Peake himself.

Teachers, register now to take part in the first round of I’m an Astronaut, Get me out of here this October!

Click to Register

The registration deadline for Round 1 is 12PM on Monday 20th July 2015.

How does it work

There will be four rounds of the competition, each running for two weeks over the course of the Principia Mission; the first round will be 5th–16th October 2015.

In each round students will log in to the site over the course of two weeks to talk and interact with members of the Astro Support Team. Students will be able to ASK questions, take part in live CHATs, and VOTE for their favourite Astro Support Team members.

Tim Peake during training at NASA's Johnson Space Center in Houston, the United States. |  Image: NASA/Bill Stafford

Tim Peake during training at NASA’s Johnson Space Center in Houston, the United States. |
Image: NASA/Bill Stafford

Live Chat with Tim Peake

In each of the four rounds, the Astro Support Team will nominate one winning school to take part in the final live chat with Tim Peake.

The winning schools will each nominate four student ambassadors to take part in the final live chat with Tim Peake when he returns from the ISS.

How do I sign up?

Teachers and Schools

Teachers can register at imanastronaut.uk/teachers the deadline to sign up for the first round is 12PM on Monday, 20th July.

Join the Astro Support Team

If you’d like to help out email Angela at angela@gallomanor.com, with where you work,  and a couple of sentences summarising your work and how it relates to Tim’s mission — when you describe your work, try to use language which a 13 year old would understand.

For more information, see imanastronaut.uk/astro-support-team.

Posted on July 15, 2015 Moderator - Josh in News | Leave a comment

Chemistry outreach secure for 3 years

I’m a Scientist is about showing the diversity of science to students and whilst we’ve always been strong with biologists and physicists until 2015 we were a little light on chemists.

RSC_LOGO_SUPPORTED_WEBSo we are especially delighted that the Royal Society of Chemistry have decided to support our project over 3 years through to the end of 2017. We will be running 9 zones across the UK and Ireland using this funding. Part of the arrangement is that we’ll be including RSC members in five of our General Zones to show school students the full breadth of science.

Rio Hutchings, from the Royal Society of Chemistry’s Outreach team, says:

We’re really proud to be involved in inspiring scientific discovery in a whole range of ways and I’m a Scientist, Get me out of here is an excellent example. It’s great for both students and for putting science communicators through their paces, so we’re very pleased to be funding events for the next three years.

The next RSC funded zone will be in November 2015; the Spectroscopy Zone. Chemists can sign up at imascientist.org.uk/scientists — Apply now, before all the places argon… Sorry.

click to apply

Posted on July 13, 2015 Moderator - Josh in News, Project News | Leave a comment

Engaging under-served audiences – talk at Bristol Zoo

Last week I was asked to speak about our experience of engaging under-served audiences. Here are my notes from which I spoke:

IAS is free online activity that connects school children and scientists. Kids go online, read scientist profiles, ask questions, take part in live chats and vote for the one they want to win.

Split into zones of 5 scientists and about 350 students. Mixture of general and themed zones. Been running since 2008 and reached nearly 70,000 students and over 1,000 scientists and engineers have taken part.

Scientists hear about it through colleagues, organisational emails and social media. They apply for the chance to take part. Tough selection process 1:4 for general zones.

Students hear about it through their teachers who sign up to take part. Over subscribed for I’m a Scientist and many teachers don’t get as many classes as they would like.

One way we do reach students who don’t normally get involved in science engagement is through product design.

Anecdotally, if a scientist visits a school, a class, 1:3 kids might stick their hand up to ask a question. With the online, pseudonymous nature of the activity we tend to get nearly 90% of the students actively participating – asking questions, chatting, commenting or voting. A side effect of this equality of voice is that not only do the quieter pupils get to ask their question but the other, louder, more confident students learn that their quieter peers do have something interesting to say. On a micro-scale this is reaching new audiences. And it is important. Not every engagement style works for everyone.

And it seems to suit some young people who don’t always get a say.

IAS is a spin off from a different project called I’m a Councillor, Get me out of here.

The inspiration for that project came out of conversations with council officers trying to interest young people in local democracy. They were doing things like inviting kids to shadow councillors for the day or they would invite students to visit the council chamber – and marvel at the majesty of our democratic overlords. The idea of an online activity was attractive to many, even if for some it just meant they weren’t going to need to clean up the graffiti left behind in the council chamber.

The why for us was that councils were trying to engage with mini-politicians, not mini-citizens. For me that was the more important democratic objective.

Over the 7 years that project ran some of the most memorable encounters were between the councillors and students from traditionally under-served groups.

I could talk about the pupil from the special school asking about the “futcha” for Bexhill or the feedback from the teacher at the PRU in Derbyshire who told us it was the first time that their pupils had ever been taken seriously by an authority figure. But it was the live chat between Nadia and a couple of Cardiff City Councillors that I remember best. The chat had been booked for 8pm by Steve a youth centre manager in Cardiff. It wasn’t the greatest of chats. Nadia was the only person online and was demanding a swimming pool for her and her mates. The two councillors patiently explained that Nadia should get together with friends and start a petition. After about 20 minutes of chat punctuated by strings of ****’s as Nadia’s choice of words got caught by our profanity filter, the chat was brought to a sudden end with Nadia typing, Nah can’t be bothered, bye bitches.

A disaster I thought.

5 minutes later Steve, the youth centre manager called me. That was brilliant he told me. Nadia had been excluded from every school in Cardiff and banned from every youth group. That night’s chat was the first time that Steve had ever seen Nadia engage with anything. Normally 1 in 5 words was a swear word but tonight in was only 1 in 20. He was ecstatic.

The point is that for some groups online communication is better than face-to-face and not just because of geography.

But having established that a project is good for certain groups it isn’t always as simple as that to get them involved.

We also look at the promotion aspect of reaching under-served audiences. Traditionally and anecdotally we’re told that science engagement projects do tend to reach the better resourced schools where kids often have plenty of privilege. Which means some schools are not taking part. But it isn’t so easy to identify them. Tied in is the issue for universities of widening participation as part of their fair access agreement required to charge £9,000 per year.

But what does widening participation mean? Apparently no two universities share the same definition (perhaps this has changed by now). We looked at a number of measures: IDACI, POLAR (participation in local area), and GCSE 5+ – when we looked at how the schools participating in IAS compared to the national profile against each of these criteria we were quite happy. We were getting a broad cross-section. And when compared to another large scale science engagement project we were performing well. But still not well enough.

There are flaws in the use of these measures. IDACI and POLAR give results based on the postcode of the school rather than it’s catchment area. The two can differ greatly. Secondly many of the schools in the poorest inner-city areas would also get greater funding through pupil premium and city-weighted education funding, perhaps allowing teachers more opportunities to bring in outside science engagement activities. Combine that with the fact that inner-city schools are also more likely to be within a short distance of a university and perhaps the inner-city schools aren’t so underserved.

So we still have a challenge on defining underserved.

We’re currently trying to work with other organisations like STEMNet to form a definition but in the meantime one of the factors we are looking at is distance. We’re looking to target schools from the poorer coastal areas in the UK.

But targeting is easier said than done.

We’ve conducted a few small experiments in recent years.

1. For one zone recently we sent 20 secondary schools in the most income deprived areas of the UK a teacher pack for the event inviting them to participate in the upcoming event. The idea was to make it as simple as possible for them to join in. Not one of them did. Not great.

2. More recently we posted a letter and some flyers to 200 primary schools in rural and coastal areas inviting them to apply to take part. The letter explained that Y5/6 was particularly important and that since they were in a remote area they might gain the most benefit. We’ve had 2 sign up. Not bad. OK

3. For a Food Zone in 2013 we looked at the schools signed up for the zone and worked out which would “qualify” as widening participation schools. There were about 5. Each teacher was called to make sure that they had received their packs and that they understood how the event worked. We looked after them, gave them special treatment. Every single one showed up online compared to a usual 2/3rds rate. It worked.

The learning we take from this is that you should look after the people already showing an interest. You get a better impact than simply going after more and more of your target. It might sound obvious, but for us we don’t have the time to call every school.
But we now realise that the time and money we might have spent sending stuff out to schools who hadn’t shown an interest, would be better spent on looking after the ones who had already shown some interest.

We’ll be repeating the specific targeting of schools in remote areas and giving special attention to those who respond. Thank you.

Posted on June 8, 2015 Moderator - Shane in Evaluation, Science Engagement | Leave a comment

March 2012 – March 2015 Evaluation Report

We were awarded a Wellcome Trust Society Award to run I’m a Scientist from March 2012 until June 2014. We later received a grant extension for November 2014 to June 2015. This report is all about our learnings in these last three years: from March 2012 until March 2015.

Our main learning points are:

WT report cover

Click here to download the full report

1. I’m a Scientist has gone from 30 zones per year in 2012, to 54 zones scheduled in the 2014/2015 school year.
2. Expanding zones to different audiences: primary school students and general public shows.
3. I’m a Scientist is a public engagement boost for scientists.
4. I’m a Scientist gets students enthused about science.
5. I’m a Scientist reaches a diverse set of students.
6. Teachers come back, but tricky to track.
7. Students ASK about cancer, animals, and life and CHAT about science, scientists and work.
8. Moving forwards, further adaptation to new technologies (such as tablets and smartphones) is important.

Click here to download the full report.

Posted on May 26, 2015 modangela in Evaluation | Leave a comment

June 2015 Moderator Job Vacancies

Hey, it’s that time again! We’re looking for moderators for our June 2015 event! I’m a Scientist and I’m an Engineer are award-winning online events allowing students (from 9 – 18) to interact with real live scientists. We’re looking for 2 moderators to work on the next event which will run from the 15th June to the 26th June 2015.

Your key responsibilities will be:

  • checking and approving questions
  • adding appropriate keywords and tags
  • logging and keeping track of questions, comments and dialogue
  • checking the site for errors and inappropriate content and usernames
  • moderating live chats
  • helping to run the site

It’s actually a lot of fun as the students (and scientists) are quick and funny and full of energy. And hey, promoting science engagement is a good thing, am I right?!

What we’d like from you..

  • You should be bright, thoughtful, pick stuff up easily, ideally with an interest in science engagement.
  • You’ll have great attention to detail (THIS IS IMPORTANT)
  • You enjoy being online, perhaps have experience in an online community.
  • The site is all built on WordPress, so if you’ve used that the techy stuff will be pretty familiar.
  • You’d be working from home, so you must also have broadband which doesn’t die every 10 minutes.

Extra bonus things we’d like, but aren’t hugely important..

  • A sense of humor
  • The ability to multitask
  • Interested in discussing your lunch

Please send a CV and short covering letter ASAP (by Monday 1st June 2015), to Emily at emily@gallomanor.com, telling us why you think you’d be a good moderator.

  • Dates: 15th – 26th June (Monday – Friday)
  • Hours: 37.5/week, 08.30-16.00 GMT
  • Pay: £7.50/hr

You can (and should) find out more about the events at: imascientist.org.uk and imanengineer.org.uk.

Posted on May 21, 2015 modemily in Event News, Science Engagement | Leave a comment

Stars, stripes, and scientists

On Monday 11th May I’m a Scientist lands in the U S of A with the very first I’m a Scientist USA event!

IAS US LOGO

We’ve had some US schools and many US scientists take part in our UK event but this is the first time we’ve run an event just for them.

Our local partner, Keep On Questioning, is looking for schools and scientists to sign up by April 13th.

So if you get any joy from explaining the world or want to attend the best crash course in science communication, sign up now.

If you want your students to meet and question a wide range of scientists from across America, then please get your class involved.

Posted on March 26, 2015 ModShane in News, Project News | Leave a comment

Moderator Vacancies March 2015

I’m a Scientist, Get me out of here, is an award-winning event where school students (age 9–18) interact with real scientists online. We’re looking for 5 moderators to work on the next event which will run 9th – 20th March 2015.

At the same time we’ll also be running I’m an Engineer, and I’m a Geoscientist.

Your key responsibilities will be:

  • Checking and approving questions.
  • Adding appropriate keywords and tags (some understanding of search and filters always a bonus).
  • Logging and keeping track of questions, comments and dialogue.
  • Checking the site for errors, and inappropriate content and usernames.
  • Moderating live chats.
  • Keeping track of your live chats and an awareness of what else is going on.
  • Taking part in discussions and giving input.

It’s a lot of fun as the students (and scientists) are quick and funny and full of energy. And hey, promoting science engagement is a good thing, am I right?!

What we’d like from you..

  • You should be bright, thoughtful, pick stuff up easily, ideally with an interest in science engagement.
  • You’ll have great attention to detail and will enjoy being online, perhaps have experience in an online community.
  • The site is all built on WordPress, so if you’ve used that the “techy stuff” should be fairly familiar.
  • You’d be working from home, so you must also have broadband which doesn’t die every 10 minutes.
  • To be always available, ready and eager to take part.

Please send a covering letter and CV ASAP (by Wednesday 18th February 2015 at the latest), to Emily at emily@gallomanor.com, telling me why you think you’d be a good moderator.

  • Dates: 9th–20th March, Monday–Friday
  • Hours: 37.5/week, 08.30–16.00 GMT
  • Pay: £7.50/hr

You can (and should) find out more about the events at: imascientist.org.uk, imanengineer.org.uk, and imageoscientist.eu. You might also want to read this, on what kind of thing we’re looking for.

Posted on January 21, 2015 modemily in Event News, News | Leave a comment