mixed methods

Tutorial: Beyond “Quant-vs-Qual”—Creating Collaborative Inquiry

ALEX HUGHES UC Berkeley JENNY LO Grammarly WILL MONGE Good Research Overview Qual and quant are so divided these days—by academic discipline, language, communities of practice, job titles. Too often, quantitative research is conflated with data science (or vice versa), and data science with optimization algorithms or simply engineering. In many organizations, being “data-driven” tends to define “data” with a narrow conception of enumeration and (mis-) conceptions about the kind of evidence that is suitable to act on. This tutorial critically examines this territory and move beyond it, empowering ethnographers to develop more interdisciplinary programs of inquiry. First the instructors review fundamentals of quantitative research and provide tools ethnographers can use to evaluate its quality and validity. Then they examine constraints and barriers to quant/qual collaboration, including time, funding, values, epistemological conflicts, organizational silos, and more. Finally, using core principles that underlie...

DIYing along with DIYers: Juggling with Scales During a Home-Improvement Research

GUILLAUME MONTAGU _unknowns As a team of researchers was asked by a French home-improvement retailer to redefine their strategy, they designed and carried out an ethnographic and quantitative research to identify new business opportunities. But no sooner had they set foot in field, they were struck not only by the richness and complexity of such ordinary activities to the point they asked themselves if these practices were even measurable? Scaling from ethnography to quantitative research was not as seamless as they expected, they had to find their way to deal with two sets of data that belong to different scales if not ontological worlds. Are these two scales really strictly separated? Can't there be a way to combine them and to make them coincide? Based on the study of DIYing practices, this case study presents an attempt to integrate ethnographic and quantitative research and the challenge of resolving the scale differences between two methodologies. From turning DIYers into numbers and vice-versa, it explores the implications...

Hybrid Methodology: Combining Ethnography, Cognitive Science, and Machine Learning to Inform the Development of Context-Aware Personal Computing and Assistive Technology

MARIA CURY* ReD Associates ERYN WHITWORTH* Facebook Reality Labs *Lead co-authors The not-too-distant future may bring more ubiquitous personal computing technologies seamlessly integrated into people's lives, with the potential to augment reality and support human cognition. For such technology to be truly assistive to people, it must be context-aware. Human experience of context is complex, and so the early development of this technology benefits from a collaborative and interdisciplinary approach to research — what the authors call “hybrid methodology” — that combines (and challenges) the frameworks, approaches, and methods of machine learning, cognitive science, and anthropology. Hybrid methodology suggests new value ethnography can offer, but also new ways ethnographers should adapt their methodologies, deliverables, and ways of collaborating for impact in this space. This paper outlines a few of the data collection and analysis approaches emerging from hybrid methodology, and learnings about impact and team collaboration,...

Ethnographic Agency in a Data Driven World

NADINE LEVIN Facebook This paper argues that ethnographers can gain increased agency in data-driven corporate environments by increasing their quantitative literacy: their ability to create, understand, and strategically use quantitative data to shape organizations. Drawing on the author's experience conducting strategic user research at a technology company, the paper explores how the ability to engage with quantitative data can increase ethnographers’ independence and autonomy within organizations, and can also up-level the role and value of qualitative research. The paper also explores how a deep familiarity with quantitative data can enable ethnographers to imbue quantitative data itself with new forms of agency, and can ultimately give ethnographers the tools to change institutions from within. With a greater understanding of how quantitative data is made and used, ethnographers can ensure that data is collected in representative ways, point out the limitations of existing metrics, and argue for new ways of measuring and...

Just Add Water: Lessons Learned from Mixing Data Science and Design Research Methods to Improve Customer Service

OVETTA SAMPSON IDEO Chicago and DePaul University Case Study—This case study provides an inside look at what occurs when methods from the data science and ethnographic fields are mixed to solve perennial customer service problems within the call center and cruise industries. The paper details this particular blend of ethnographic practitioners with a data scientist resulted in changes to design approaches, debunking myths about qualitative and quantitative research methods being at odds and altering team member perspectives about the value of both. The project also led to the creation of innovative blended design research and data science methods to discover and leverage the right customer data to the benefit of both the customer and the call center agents who serve them. This paper offers insight into the untold value design teams can unlock when data scientists and ethnographers work together to solve a problem. The result was a design solution that gives a top-performing company an edge to grow even better by leveraging the millions...

Designing for Interactions with Automated Vehicles: Ethnography at the Boundary of Quantitative-Data-Driven Disciplines

MARKUS ROTHMÜLLER School of Architecture, Design and Planning, Aalborg University Copenhagen, Denmark and Shift Insights & Innovation Consulting PERNILLE HOLM RASMUSSEN School of Architecture, Design and Planning, Aalborg University Copenhagen, Denmark SIGNE ALEXANDRA VENDELBO-LARSEN School of Architecture, Design and Planning, Aalborg University Copenhagen, Denmark Case Study—This case study presents ethnographic work in the midst of two fields of technological innovation: automated vehicles (AV) and virtual reality (VR). It showcases the work of three MSc. Techno-Anthropology students and their collaboration with the EU H2020 project ‘interACT’, sharing the goal to develop external human-machine interfaces (e-HMI) for AVs to cooperate with human road users in urban traffic in the future. The authors reflect on their collaboration with human factor researchers, data scientists, engineers, experimental researchers, VR-developers and HMI-designers, and on experienced challenges between the paradigms of qualitative...

Gaming Evidence: Power, Storytelling and the “Colonial Moment” in a Chicago Systems Change Project

NATHAN HEINTZ Systems Change Consultant Case Study—In 2016 The Chicago Community Trust (“The Trust”), a local Chicago foundation, partnered with Roller Strategies (“Roller”), an international professional services firm, to deploy an innovative mixed-methods approach to community-driven social change on the South Side of Chicago. This partnership convened a diverse group of stakeholders representing a microcosm of the social system, and launched a project with the aim of developing resilient livelihoods for youth aged 18-26 in three specific South Side neighborhoods. Roller designed and facilitated a process through which the stakeholder group scoped, launched, piloted and prototyped community-driven initiatives. While innovative and successful by some metrics, the project had its challenges. The convening institutions and their staff were often perceived as “outsiders” and “experts” without intimate local knowledge of the social challenges they were attempting to address. This dynamic played out in complex power...

ReHumanizing Hospital Satisfaction Data: Text Analysis, the Lifeworld, and Contesting Stakeholders’ Beliefs in Evidence

JULIA WIGNALL Seattle Children's Hospital DWIGHT BARRY Seattle Children's Hospital Case Study—Declining clinician engagement, increasing rates of burnout, and stagnant patient and family experience scores have led hospital leadership at Seattle Children's Hospital to submit requests to a data scientist and an anthropologist to identify key themes of survey comments and provide recommendations to improve experience and satisfaction. This study explored ways of understanding satisfaction as well as analytic approaches to textual data, and found that various modes of evidence, while seemingly ideal to leaders, are hard pressed to meet their expectations. Examining satisfaction survey comments via text mining, content analysis, and ethnographic investigation uncovered several specific challenges to stakeholder requests for actionable insights. Despite its hype, text mining struggled to identify actionable themes, accurate sentiment, or group distinctions that are readily identified by both content analysis and end users, while more...

Below the Surface of the Data Lake: An Ethnographic Case Study on the Detrimental Effect of Big Data Path Dependency at a Theme Park

JACOB WACHMANN ReD Associates ANDREAS JUNI ReD Associates DAVE BAIOCCHI ReD Associates WILLIAM WELSER IV ReD Associates Case Study—This case-study details how a team of anthropologists and a team of data scientists sought to help a Middle Eastern theme park make use of their big data platform to measure ‘the good customer experience’. Ethnographic research within the theme park revealed that visitors yearned to bond with the other members of their group, as they rarely got the chance during their busy everyday lives back home. However, trying to build a measurement of how the theme park delivered on bonding – through the development of a ‘bonding index’ – turned out to be unfeasible, because the big data platform focused on capturing operational data. The decision to focus on operational data had unintentionally created a path dependency that made the big data setup unfit for answering some of the theme park’s most fundamental questions. This is a problem ReD Associates has observed across clients and to solve it this...

Revitalising Openness at Mozilla: A Mixed Method Research Approach

RINA TAMBO JENSEN Mozilla Case Study—This is a case about how Mozilla, the open source browser company, set out to reconnect with ‘collaborating in the open’ to regain its competitive advantage. This case describes how a multi-disciplinary research team used ethnographic, market, and data analysis to articulate and clarify the problem, and build a strategy towards revitalizing Openness at Mozilla. It will aim to prove that the subsequent change achieved could only have been accomplished by a mixed method research approach. And importantly show, how the team used data to prove the distribution of findings, coupled with ethnography to shine light on the why and how of those findings. The case study will do this by discussing the key insights and how these fueled recommendation and subsequent change in the organisation. The project presented many problems: from convincing stakeholders of the need to fully explore the problem, to connecting widely different research methods and gleaning insights that built strongly on all strands...

Data Science and Ethnography: What’s Our Common Ground, and Why Does It Matter?

by TYE RATTENBURY (Salesforce) & DAWN NAFUS (Intel) As EPIC2018 program co-chairs, we developed the conference theme Evidence to explore how evidence is created, used, and abused. We’ll consider the core types of evidence ethnographers make and use through participant observation, cultural analysis, filmmaking, interviewing, digital and mobile techniques, and other essential methods, as well as new approaches in interdisciplinary and cross-functional teams.1 We’ve also made a special invitation to data scientists to join us in Honolulu to advance the intersection of computational and ethnographic approaches. Why? One of us is a data scientist (Tye) and the other an ethnographer (Dawn), both working in industry. We regularly see data science and ethnography conceptualized as polar ends of a research spectrum—one as a crunching of colossal data sets, the other as a slow simmer of experiential immersion. Unfortunately, we also see occasional professional stereotyping. A naïve view of “crunching” can make it seem...

A Mixed Method Approach for Identifying Emerging Fields and Building Collaborative Teams: Leveraging Network Ethnography to Design Experimental Interventions

THERESE KENNELLY OKRAKU Microsoft VALERIO LEONE SCIABOLAZZA University of Florida RAFFAELE VACCA University of Florida CHRISTOPHER MCCARTY University of Florida Rapid innovation in science and technology has led to the development of new fields that transcend traditional disciplinary boundaries. Previous studies have retroactively examined the emergence of these fields. This paper outlines a mixed method approach for using network ethnography to identify emerging fields as they develop, track their evolution over time, and increase collaboration on these topics. This approach allowed us to simultaneously analyze organizational trends and gain an understanding of why these patterns occurred. Collecting ethnographic data throughout the course of the study enabled us to iteratively improve the fit of our models. It also helped us design an experimental method for creating new teams in these fields and test the effectiveness of this intervention. Initially, organizational leaders were wary of using a network intervention...

The Domestication of Data: Why Embracing Digital Data Means Embracing Bigger Questions

DAWN NAFUS Intel Corporation The EPIC community has been wrestling with ways to integrate quantitative and qualitative methods in light of the increasing role that digital data plays in business practices. Some focus on methodological issues (digital data as method), while others point to the consumer value in data products (data as thing in the world). This paper argues that “digital data as method” and “digital data as thing in the world” are becoming increasingly intertwined. We are not merely witnessing ethnographers’ haulting embrace of digital data, but a wider process of the domestication of data, in which we, alongside the people we study, are participants. The domestication of data involves everyday situations in which ordinary people develop their own sense-making methods—methods remarkably similar to ethnographic knowledge production. In this way, the domestication process tightens the connection between data as thing in the world and data as method. I argue that seeing the interconnection gives us the...

Unleashing the Power of an Analytics Organization: Why a Large Financial Institution Used Ethnography to Transform Analytics

by CHRISTINE BIRTEL (Senior Vice President), JASON PAJTAS (Vice President) and MICHELLE GREEN (Vice President), Customer Experience Insights, Wells Fargo Large organizations have been on a quest to harness the power of big data to inform and drive all kinds of decisions—from finance and fraud prevention to product development and marketing. Organizations from the U.S. government, to retailers, to major financial institutions are appointing Chief Data Officers (CDOs) to evangelize the power of big data. The path of creating a data-savvy organization, however, has complicated turns and roadblocks. Starting with the technological foundation for analytics, there exists a dizzying array of platform and software solutions all claiming to be best-in-class. When success drives an organization beyond this foundation, evolving an existing culture and data infrastructure around things like cloud, big data, machine intelligence and block chain can be an overwhelming challenge that seems insurmountable. Beyond infrastructure challenges,...