The Automation of Qualitative Methods

Share Share Share Share Share


Bauhaus by cdschock via flickr


Anthropology and its methodologies cannot easily be automated. However, both design and engineering based organizations are attempting it. I argue that this is based in part on historic legacy systems, a misunderstanding of the ethnographic toolkit, and an over-reliance on the principles of Bauhaus, Six Sigma, and Science Fiction.

Quantifiying and Automating the Qualitative

After interviewing at several other engineering focused companies in their User Experience groups, I recently interviewed for a job at a renowned design firm. The design firm advertised for a "Director of Insights and Strategy," a job I’m well suited for. However, after I travelled to their offices, gave a presentation and spoke with them, it became apparent that what they really wanted was probably just a Research Manager (e.g. someone to provide a checklist of conventional methods that can be replicated, and someone who would guide others to use them as well). Prior to my arrival, the company had said that they wanted "innovation and creativity," but when confronted with the prospect of that and what it meant, they reverted in the interviews to requiring me to list traditional methods—methods which they had initially claimed weren't working, and methods which they appeared to quantify as a mark of assessing my expertise.

The repeated pattern I’ve seen in design firms, start-ups, and even some established larger companies, uses automation principles in its enactment. Initial correspondence may be with the hiring manager, but interviews are set up by an HR person or calendar bot and scheduled via email. The hiring manager delegates the interview to a junior employee, who usually has no scope of project and little experience in a range of project types, and who most often comes from an entirely different discipline. Interview questions are formulaic, relating to quantifying my methods, and streamlining particular methods—without regard for the broader methodological context of when these methods may or may not work—or why—or in what context. I express curiosity about the questions the interviewers ask, and bring up the importance of a discovery or definition phase to determine appropriate methodology. I also query how what they are proposing to research fits into a broader realm. These questions are not interesting to these interviewers, who prefer to guide the interview with role-playing scenarios, and I have been asked to "geek out" on terminology and methods. In these cases, my lines of qualitative inquiry are not identified as such and are misinterpreted as naive and uninformed, rather than the 'probe and discovery' method that I am using. Although some interviewers allotted me more time, in many situations the interviewer spoke for exactly 25 minutes and offered me 5 minutes to ask questions, with a sharp termination at the 30 minute mark, leaving little or no time for my questions to be answered.

Based on repeated exposure to this pattern, it has occurred to me that the disciplines of engineering and design are trying to find ways to codify, repeat, and, more critically, automate the process of hiring anthropologists (and others), as well as to automate their research methodology.

The other pattern that I have noticed is that corporate User Experience Research groups are frequently headed by a male with an upper management title who holds a degree in a field such as literature, arts, business, economics—but not anthropology, sociology, or psychology—and who has built up a group of much younger practitioners generally with design or other non–social science backgrounds. Occasionally, I was asked to interview with an early career cognitive scientist (a discipline mostly is concerned with an individual's brain function). Thus, as I seek my post-doctorate work 'home,' I’m becoming increasingly concerned, in a broad sense, about the role of anthropology in the design and technology industries, and where my skills will be authentically valued.

I consulted former colleague and design anthropologist Dr. Susan Squires, and asked for her take on why I was repeatedly being asked to list methods and to "geek out" on terminology in these research interviews. She observed that those who are measuring my worth by lists of methods may be confusing data collection with research, and that to them, the more methods one has or can discuss, the better and more senior they will be at what they define as "research." For designers who are not steeped in research, and lack a research degree, methods may be their only metric for data collection, so data collection is their definition for research, rather than methods merely being a part of what research encompasses.

These research managers seem to have a need to quantify methods or other attributes. For the most part, ethnography is omitted from the conversation, or described as a single method rather than as a toolkit of methods. The people in control of interviews do not understand the basics of ethnographic methodology and what anthropology does, or can do for them, instead assuming it to be a part of a discipline that can be streamlined, made 'lean,' easily quantified, and lumped in with their other methods.

Bauhaus, Six Sigma, and Science Fiction

In both design and engineering, there is a similar trend to ‘simplify' and 'reduce to basic principles,' and at times, both share a 'least common denominator' way of thinking. In design, the idea of 'simplify' has worked well visually in terms of clearing up visual space, but it has done nothing to handle complexity.

I spoke about this with Daniel Drennan El-Awar, a graphic designer and former professor in graphic design and media design. He reminded me that designers today are still influenced by Bauhaus, an experimental German post-World War I modernist movement founded on the idea of reduction and simplicity of both form and color palette, which additionally incorporated white space. Building on that notion, as a design movement from the early part of the 20th century, Bauhaus could be considered a form of automation, which was also debated at the time.

Today, we still see the Bauhaus influence in our furnishing movements, architecture, and other areas of design that rely on the notion and visual language of clean, white space, and simplicity as aspirational qualities—even though for the most part Bauhaus did not produce livable spaces that endured for communities of people. Its reductionist principles are simply unsuited to address social complexity.

Meanwhile, the idea of Six-Sigma has largely influenced User Experience and User Interaction design. Developed by an engineer at Motorola as a process improvement strategy, Six Sigma was adopted across a wide range of industries, especially business. Six-Sigma relies on two methods: DMAIC (Define, Measure, Analyze, Improve, Control) and DMADV (Define, Measure, Analyze, Design, Verify). Much of the focus of Six Sigma is on quantitative measurement, and more recently, Six-Sigma has been adopted as a way to streamline UX design, by reducing it to standardized principles, and applying these across all forms of design. Organizations wanting a fast formula for 'better' processes can easily be persuaded to include the reductionist rules and methodology of Six Sigma, which advocates rigid methods that are perceived to be a reliable formula. When designers and engineers work together in engineering based organizations, any process that provides quantified results, which support a quantitative, mathematical, and reductionist approach, seems to be preferred.

It makes logical sense that Six Sigma's process would derive from an engineering mindset, and it makes logical sense that when faced with users and their needs, engineering would look for an engineering solution to solving said 'problems' with regard to people.

Engineering, in particular as it is practiced in Silicon Valley, also has an aspirational vision described in Science Fiction, and which is built visually on the design forms founded in the 1950s and 1960s (based on Bauhaus). Although we've moved along in years, many of our design and engineering visions are inspired by a 60-year-old imagination of the future, which is often described in a context 300 years ahead of where we are now.

Where This Leaves the Ethnographer and Other Qualitative Researchers

The current path of User Experience design (and by association, User Experience Research) has been deeply co-opted by Six Sigma principles, by designers and design firms steeped in Bauhaus training, and practitioners required to function within the scope of an engineering industry that prizes mathematics and quantification as their only acceptable metric. If ethnographers and other qualitative researchers join these companies in a research capacity, we are expected to embrace three 'beliefs': 1) design 'simplicity' and Bauhaus; 2) a Six Sigma method of quantification as the only method for measuring results; and 3) an engineering vision derived from science fiction, which provides a working solution that only requires measurement of effectiveness because it's functionality has been previously demonstrated to 'work' in Science Fiction narratives.

Ethnographic praxis does not easily lend itself to any one of these disciplines or methods precisely because people are complex, take agency in their own ways, and are part of a reality that is the now: the nested and interrelated systems in which we live and exist, overlap, fragment, spiral, swirl, and intertwine with each other, or miss and do not overlap at all.

All three of these dominating 'beliefs' from design, manufacturing, and engineering lend themselves easily to forging automated processes, where rules can be put in place and enacted in projects that fulfill the vision of the aspirational future from both design and engineering. However, the outcomes from these over-focused, rigid, and brittle processes will be vulnerable, as systems become inflexible to change and human agency. We are up against a triad of methodology and beliefs that were created for things, but are now being applied to people. People are creative and prefer to take agency when we are trying to solve problems, cooperate, and engage with the world. The metrics-based 'automated process' pattern is building a world that will be easy to automate initially, but that may be—and already is in some instances—ineffective and potentially dangerous as brittleness and inflexibility increases, and human agency decreases, limiting our options and choices for problem solving and cooperation.

As EPIC practitioners, we must consider how to offer our tools, knowledge and skills to organizations and industries that value what we have to offer, and that are not yet automated, or which may be but, after many mistakes, now understand the grave nature of getting it wrong. We will do best in companies who have expanded their belief systems to include us, and our toolkits, in order to create more flexible, adaptive systems, which will preserve and encourage the entire range of human creativity.

Photo: "Bauhaus" by cdschock (CC by 2.0) via flickr.

sally applinSally A. Applin is a Senior Researcher based in Silicon Valley, examining the changing relationship between humans and algorithms, the impact of technology on culture, Maker culture, leading technologies, and the outcomes of network complexities as modeled by PolySocial Reality (PoSR). Dr. Applin earned her PhD in Anthropology at the University of Kent at Canterbury, UK, working with the Centre for Social Anthropology and Computing (CSAC). Sally is an Associate Editor of the IEEE Technology and Society Magazine, Associate Editor of the IEEE Consumer Electronics Magazine (Societal Impacts Section), a member of IoT Council (a think tank for the Internet of Things (IoT)), and a board member of the Edward H. and Rosamond B. Spicer Foundation.


The Rise of the User and the Fall of People: Ethnographic Cooptation and a New Language of Globalization, Shaheen Amirebrahimi

The De-skilling of Ethnographic Labor: Signs of an Emerging Predicament, Gerald Lombardi

The Future of User Research, Apala Chavan

Ethnographers, Bearers of Bad News, Sam Ladner

  5 comments for “The Automation of Qualitative Methods

  1. Sally I agree with your concerns and complaints about reducing research to automated and repeatable methods. I would disagree with putting design (Bauhaus) or any other design pedagogy in the same bucket with Six Sigma. You are comparing apples and oranges.

    However the digital design that you talk about in start-ups or big corporates is mostly led by HCI, IT or MBA folks, none of them have a design or Bauhaus education, even if they are designing or directing the design verticals. In my experience it is not due to design education but lack thereof. Their hallmark methods such as Six Sigma and Lean are generally used for production, whose primary objective is to be efficient. Design processes are not about efficiencies, they are intuitive and generative. Unfortunately the design practice of physical objects is very different compared to the field of UX where the user experience has been reduced to screen experience, again due to lack of proper design training. The field is also a wild wild west, many professionals come san experience or with very little. Hence the challenge that you face is also faced by trained designers of physical objects.

    On the other hand your observation about design, as simplifier of complexity is true, to an extent. Design is about solution finding, which doesn’t take place by over complicating things. A good designer is able to understand the complexity and reach the crux of the issue that needs to be solved. Simplification doesn’t mean lack of complex understanding, it can also mean acknowledgment of complexity and ability to break it into workable parts and pieces.

  2. Thank you for your comments.

    Let me unpack this a bit more for you. What I am referring to with both Bauhaus and Six Sigma are rulesets that inhibit human agency. At the time, Bauhaus was considered strict in its methods, and a form of automating design, reducing color palette and form. These are not “apples and oranges,” rather, these are different movements at different times with similarities, if one thinks laterally about them.

    Bauhaus is an reductionist paradigm. This means that things got left out. Ornaments, indicators, decorations, warmth, etc. What came to the forefront of it was simplicity and (efficiency by proxy): in architecture, in construction, in process. Six Sigma has a similar value proposition. Both can be argued to have similar objectives and goals when applied to their various problem sets.

    Cultural soup spreads over generations. Bauhaus is in things. It influenced a lot of architecture in its time, which is still residual in institutional buildings such as universities, it permeates typefaces and graphic design methods, and it greatly influenced much of the industrial design objects we use today. Those in power in these organizations may lack a “design” education as well, but they’ve been influenced by their culture. We can’t escape that.

    For context, I have degrees in fine art and design, in technology, and now in anthropology. I have worked in design/UX and UE research for many years in an applied context. I have seen a lot of change and a lot of the same problems circulating again and again.

    Six Sigma has infiltrated management of software design, particularly UX, and has changed the way that organizations which produce UX are held accountable. Quantification matters more now than ever to this particular trend, and things that were deemed “decorative” have been reduced.

    Simplifying complexity is a dangerous game with humans. We like to make choices from options and sometimes the choices we want to make aren’t available as options because those in the “lean” paradigm removed them for efficiency, or cost, or aesthetics, or because we think that we can break them into “workable parts and pieces,” or whatever. When simplifying, often some or many of the “workable parts and pieces” are left out, or reduced, and the whole framework of complexity is not accounted for. This is where the danger lies, and where good qualitative methods and practitioners can preserve the integrity of the structure.

  3. Hi Sally, great text.
    I am a designer in a financial technology company, which is growing very fast. So far we’ve relied on our designers’ research skills to do all our qualitative research, but we’re looking to build a more structured research organization moving forward. We’ve never done this before and are very open to creating the best environment for research to thrive. What would your advice be for us in this case? How can a research professional & her team be most successful in delivering value to the rest of the company?

  4. Hi Sally, very interesting read. Enjoying it (although very late to the party – just joined EPIC today).

    On one section, I have to question or challenge you thought, as I feel you’re making a generalisation / assumption.

    Re “The current path of User Experience design (and by association, User Experience Research) has been deeply co-opted by Six Sigma principles, by designers and design firms steeped in Bauhaus training, and practitioners required to function within the scope of an engineering industry that prizes mathematics and quantification as their only acceptable metric.”…

    I’ve been working in the professional technology services industry for over 15 years (in Melbourne, Australia), and I can tell you Bauhaus and Six Sigma has never come up, I and many of my industry colleagues never trained in them and they don’t drive how we lead and manage and recruit UX research practitioners. The challenge in the professional services industry is: time! As our clients pay high day rates for services, they expect thought leadership, and speed to delivery.

    Having said all this though, I do concur that some people don’t truly understand the depth of ethnography and undervalue the skills that such practitioners can bring to the table.


    • I forgot to add – it’s this need for speed to delivery that forces the hand of researchers to come up with efficient cookie cutter ways of applying research methods quickly and synthesising even more quickly. I gather this has been the catalyst for the ResearchOps movement, that and the need for more consistency of research methods across practitioners.

Leave a Reply