This is by way of a summary of the 2 sessions I initiated on the topic of automation in market research. The first being a webinar as part of NewMR’s festival of MR at the end of February. And the second being a panel discussion at the MRS national conference. The pretext for wanting to have a discussion on automation was because of the way it was talked about as inevitable and desirable. And alongside it the idea that the number of people in research is set to decline by as much as half. But the specific trigger was reading Contented Dementia by Oliver James who explicitly warns against asking Alzheimers patients any questions. Because they are no longer laying down new memories and are unable to construct an answer (which is distressing for them). This reminded me of the cardinal principle of elicitation – that when you ask someone a question they don’t retrieve the answer for you they have to construct it. And depending on context, they may also tailor their answer to the appropriateness of the context and how they choose to answer who is asking the question. So question and answer is a far from trivial matter and it seemed to me that automation was likely to be treating it far too simplistically. Oliver James was due to give a keynote at the MRS conference in March of this year. So I got in touch with the MRS and asked if it would be possible to involve him in a panel discussion as well and to gather a panel around him to discuss the topic of automation. Which they and Oliver readily agreed. In this post I want to highlight some of the main points I took away from the discussions. You can hear the recording of the NewMR debate here. And you can read a review of the MRS panel discussion here. Unfortunately, the second session was not recorded and although I had a recorder to hand I was too flustered as the chairman to remember to switch it on. Sorry! butI hope the following post gives a flavour.
This was more of a debate with Ray Poynter who had written a report about automation as chair of the automation team if I can call them that. Fiona Blades of Mesh Experience whose company uses data sent via mobiles to measure customer experience. Steve Phillips the founder of Zappistore one of the leading agencies involved in automation. On my team we had Natalie Geddes director at the retail specialist agency ABA and Rachel Lawes of Lawes Consulting, a specialist in semiotics and anthropology.
The point was made that research has always used scale and automation to do its work. But that the last few years have seen a stepchange in how much automation is being used. Good practice dictates that quantitative surveys should be piloted and this practice was becoming less common meaning that surveys weren’t always properly understood and could generate rogue results when questions were understood and answered at cross purposes.
Steve Philips gave a robust defence of Zappistore’s methodology. Arguing that the same questions were used over and over in many different surveys so had been more effectively piloted than most ad hoc surveys. It was the repeatability of the questions which meant that clients could have confidence in them and the questions were expected to be reviewed every 18 months as language use evolves. His argument is that the only automation which is sanctioned is where something can be done by a machine. And despite the influx of capital from venture capitalists it is researchers who are making the decisions about what can and cannot be automated. And frequently it is researchers who are deciding not to automate or use an automated survey as a component in a mixed mode methodology which involves other forms of data collection other than automated surveys.
Rachel Lawes was concerned about the loss of context. That the asking of a research question was more about where it was being answered than who was answering it. Online research usually doesn’t take the time to find out where people are when they answer the question. But the automators argued back that where this was critical the question would be asked. And that the variety of contexts where the questions were answered would even out.
Natalie Geddes felt that all research couldn’t be reduced to online there would always be a need to interview people in location to see how they shopped and how they used products. So online surveys were a useful part of the repertoire but couldn’t replace all of it. ABA were also running surveys in which thematic apperception questions were being asked and however automated the collection of these the analysis still had to be done by a human being.
Which brought on a discussion about the extent to which AI was deployable in analysis. It was felt that AI had a long way to go and that although data could be sorted and indexed that open questions still had to be analysed by hand. The issue was when AI would be sufficiently sophisticated to be allowed to work alone rather than as an amplifier of a human analysts work. We’re not there yet.
Fiona Blades pointed out that the automation of data once collected had made a huge difference to speed of turnaround and benchmarking. And this had allowed marketers to understand the nuances of multimedia campaigns and how these were working.
This discussion was much more focussed on the issue of elicitation and the creation of answers. Panellists were Oliver James the psychotherapist, Steve Phillips of Zappistore, Anjali Puri Global head of Qual at the research agency TNS Kantar. And Dr Mariann Hardey of the Advanced Research Computing centre at Durham Business school. And I chaired the discussion.
Oliver explained his understanding of elicitation and the panel agreed a common definition
Elicitation – has to do with all primary research involving actions taken by the researcher which will affect or incite a response from the participant. And what the participant produces is something which emerges from the elicitation -it is not simply retrieved.
and Steve offered a definition of automation which the rest of the panel accepted.
Automation – any task which a computer does instead of a person.
Oliver’s point is that the asking and answering of questions is a human activity. And that each party adopts a persona from which to answer the question. He had just been speaking about the way David Bowie dealt with his own mental health issues by distinguishing between his separate personas of David Jones, David Bowie and Ziggy Stardust. Who is being asked the question and who is answering it? The researchers were less concerned about who was answering the question because that emerged organically from context.
Once again Steve argued that automation represented a step change, not a seachange, which makes research accessible to many more companies than ever before. For example surveys could be timed to the appropriate time of day when it might otherwise be difficult to arrange for human researchers to be able to conduct interviews.
Anjulia Sharma argued that machines could be applied to do elicitation. Even if she did not think that AI was capable of replacing humans to do analysis. Mariann Hardey – present by video screen warned of the challenges offered by social data when brands had different meanings online when used as social currency. This was likely to affect how brand studies performed. Automation could not be applied to everything even if social media harvesting was now automated.
Mariann Hardey – present by video screen warned of the challenges offered by social data when brands had different meanings online when used as social currency. This was likely to affect how brand studies performed. Automation could not be applied to everything even if social media harvesting was now automated.
There was an opportunity to vote at the end of the discussion – and those who believed that automation could coexist with elicitation vastly outnumbered those who were troubled by it – 96 filed out through the Ayes door and 6 via the Noes door!
There is no doubt that automation is on a massive growth curve at present. But I was pleased that in the 2 discussions we were able to rehearse how complex research is and what automation needs to achieve before you can automate ALL research.
For those who are curious and pick up this post in time. NewMR have yet another talk about automation on Friday March 31st which you can find out more about here! Ray is telling us when automation should be applied to research. Here’s the link.