This paper explores ways in which ethnographic impact in a large technology corporation is perceived, re-defined, and recognized – by both practitioners themselves and corporate stakeholders. The authors trace a history of ethnographic successes and stumbles, and ways they have confronted a strong usability paradigm that has shaped organizational assumptions of impact and value for product research. They then identify ways in which contextual analysis of their own practice in the corporation led to the successful creation of a strategic engagement model for ethnography, resulting in its growing influence. Through critical analysis of the conditions of influence in their own organization, the authors’ propose some broader frameworks for ethnographic impact and raise some questions for the EPIC community regarding business value, ethnographic identity, and organizational authority.
INTRODUCTION
We often describe our roles as ethnographers as ‘sense-makers’ – deciphering and isolating patterns of meaning within complex landscapes of people, things, places and the sociocultural dynamics that define our various interactions. In an applied business context, we believe our ability to “make sense” of our products/services, customers, or organizations offers adequate business value to justify the costs of our employment (Robinson, 1994). Over the past few years at EPIC we have engaged in various discussions about our roles as sense-makers in organizations, “who are we?”, the approaches we take to sense-making, “how?”, and the types of problems we try to make sense of, “why/what?”. Here we want to examine the “so what?” or “why does this matter?” question about the work that we do – which is often the final analytic filter we apply to our data in order to extract applied value. When we hold a business lens of scrutiny up to ourselves, how do we fare? We want to have an open conversation about how we can trace – if not quite measure—our impacts, identify the types of impacts we can have, and think critically about how we can grow them over time both as individuals and as a practice.
As a community of practitioners, we share ways in which we create impact within our organizations by leveraging our craft at building deep knowledge of people within the contexts of our applied foci. However, because we are scattered across a variety of contexts of practice—inclusive of consultancies, non-profits, private companies, government agencies and universities—we must take different approaches in turning that knowledge into a branded commodity that gets used to create value in our diverse organizational contexts. In this paper, we endeavor to reflexively assess our own failures and successes in creating business impact within the specifics of our own organizational context.
We begin by tracing the evolution of ethnographic practice within the specific landscape of expectation and perceived value of Microsoft, and identify factors that shaped the relative impacts of ethnographic projects. We then look at how we turned our analytic lens on our own roles in the organization, and applied our skills at opportunity analysis to create a strategic engagement model for ethnography within the Microsoft product development cycle and organizational terrain. As we have raised the stature of ethnography in our organization, we have seen an evolution of increasing impact of our work. While this propels us forward as a community of practice and individual professionals, we end by raising familiar but not yet deeply explored questions about the interplay between ethnographic identity and organizational authority. Our goal is to emerge with some frameworks of ethnographic impact that you can apply and stretch and enrich within your own contexts of practice. We believe that, as a community, we can mature our business effectiveness by working together to better define what it means for us to create impacts with our work, and anticipate the paths down which those impacts will take us.
SITUATING OUR OWN SENSE-MAKING
“Usability engineering has been an extremely influential approach to technology design. … Its philosophy of specify-design-test-redesign etc. is both action oriented and pragmatic.”
(Dillon, 2000)
We begin with situating our own context of praxis, sharing some insights of the historical trajectory of research and ethnography within Microsoft and ways in which conceptions of research ‘impact’ have been viewed and measured here. Microsoft operates on a traditional software development cycle with four distinct phases: Phase I – Planning the Product, Phase II – Designing and Building the Product, Phase III – Testing the Product and Phase IV – Releasing the Product.
Research to guide the strategic direction of a product has traditionally fallen within marketing at Microsoft, which is focused on selling products rather than building products. Market research usually occurs during Phase I, when marketing conducts large quantitative surveys to determine what the main goals of the product should be, assessing people’s broad wants and needs or what drives them to buy, but not necessarily specific to how people use a product.
Research within product development has its origins in Product Testing where Testers try to find potential problems with the software, commonly called ‘bugs’. In the late 1980s the emerging field of usability was being embraced widely across the technology sector, most technology companies hiring usability engineers and investing in usability labs (Dumas, 2007). Across the industry, there was increasing awareness of the value of having target customers utilize the software in order to identify and ‘fix’ bugs so that products worked better but were also easier, more efficient and more pleasant to use (Nielsen, 1993). Within Microsoft the usability engineering cycle for product research is an iterative process, whereby software code is written, a usability engineer tests that code with ‘users’, issues are identified, implications for design are established and then the highest priority issues are addressed in the next update to the code. This has led to a specific model of impact for usability at Microsoft: to list the recommendations for changing a product, and to directly show where changes were successfully promoted and implemented in the shipped product. Hence, ‘good’ product research findings began to be defined as those that were the most concrete and ‘actionable’ (Diagram 1.1).
In relation to the product cycle, usability was initially employed during Phase III, to ensure there were no major problems before releasing a product. It was recognized, however, that starting product research at this time did not allow time to actually make changes to the product before it was released. This stymied a usability engineer’s ability to make impact to the product. Therefore usability researchers began conducting studies earlier in the product cycle, in an attempt to catch user-interface difficulties earlier so that they could more effectively impact the product. Kent Sullivan, a Microsoft veteran who began his career at Microsoft in the late 1980s as a usability engineer, explains this process, “The history of usability is trying to row upstream. Water is flowing downstream as the product cycle is unfolding while we were trying to row backwards in time and figure out how to have impact earlier and earlier in that cycle.”
As researchers began to be embedded in the product cycle earlier, team members began to ask questions of them that traditional usability methods did not well address: How are people using our products in the real world? Where do they use our products and why? How do these things differ market to market? While market research gave insight into some elements of these questions, like how many people used a particular product, or what people said were their desired improvements, it did not get to a granular level of how, where and why. Methodologically, both contextual inquiry and ethnography were explored to help fill the recognized gap. Contextual inquiry is still utilized as a method by many usability engineers today throughout the product cycle (Beyer & Holtzblatt, 1998). However, there was still a need for a more encompassing approach during Phase I, allowing for a broad perspective to help team members deeply understand, as well as connect with their target customers. Therefore, similar to usability researchers, a new type of practitioner with a strong sense of identity related to a method was brought into Microsoft product development, the ethnographer.
CONFRONTING LEGACY MODELS OF IMPACT
While there was an identified need for building deeper customer understanding, there was no identified model by which an ethnographer should integrate their work into existing structures and processes. Ethnographers have now been on staff across Microsoft for almost 10 years, numbering eight specialized practitioners across various product teams, three specialized practitioners within Microsoft Research, and a small handful that do not have the title of anthropologist or ethnographer, but view their ethnographic praxis as pivotal within their roles. While every large product team at Microsoft has usability engineers (a 200+ person community across the company), there are relatively few teams that choose to employ ethnographers (although some teams do hire consultancies to engage in ethnographic inquiry for them).
Through the years of having full time ethnographers, an extensive number of ethnographic investigations have been conducted covering a wide range of goals. Some of these have been exemplar in building insights and creating widely recognized value. Some investments have had narrow, but significant, impact to lay a foundation for change. Other projects have been supported and loved by the ethnographic community, but have been disappointing in the minimal traction they gained. So what has worked, what hasn’t worked, and why?
Ethnographies That Had No Home
Looking back it is easy to see that our biggest failures were places where our work was ‘good’ in its own right, but outside the expected metrics. As researchers sitting within product development, we faced assumptions that our integration into the product cycle and our impact on the product would follow a similar trajectory as usability. But our work does not always conform to the model of impact that usability has forged. In his paper discussing the disconnect between ethnography and traditional HCI practices, Dourish points out a common response to ethnographic work: “ ‘ it’s all very interesting, but I don’t understand its implications for design’ ” (Dourish 2006). For us, the least successful projects have been those that we liken to islands: pieces of research that in and of themselves are interesting and insightful, but fall outside the context of what is going on with the larger product team. As ethnographers projects like these are tantalizing for us, in their wide open prospects, as well as the chance to unveil strategic business opportunities that were otherwise unnoticed. Our handicap in these cases has not been in the validity or quality of the research, but in varying degrees of success in situating the research within strategic business paradigms.
There are two reasons why particular projects have not been as successful as other ethnographic projects at Microsoft: 1) when a project does not appear to have direct relevance to the short or medium term business goals; 2) when there is no team already in place to act on data that we bring back from the field.
Too Far Outside the Product Cycle – It is common for ethnographic methods to be used to explore new opportunities, helping to determine whether an investment should be made. However, at Microsoft, when ethnographic projects are too far removed from the current product, it can make it difficult for our peers to find worth in the investment. In one series of investigations management was considering what it would mean to take a standardized global product and add elements to make it look at feel as if it were developed for a specific market. The ethnographers were tasked with exploring cultural nuance with this lens, while the rest of the product team was working on the next standardized version. Imagine the clash when two ethnographers give a presentation on how the computer is or is not relevant to the lives of middle income families in India to an audience that must return to their office and focus on building for US and Western European markets. While our management was looking at long-term strategic value, our colleagues did not necessarily understand how the project related to what they were doing today.
No Organizational Target – When we do uncover strategic business opportunities, it can be difficult to find someone who has the power to take immediate action on them. This is different than our usability counterparts because they are mapped to a feature team and asked to provide direct feedback on a plan or product that has already been produced. New business opportunities often do not have a direct organizational target, nor people at the ready to implement the feedback. Due to the legacy model of impact we must show how our research impacts current products, therefore if there is no one within our own division ready to act on our recommendations, we have been forced into the uncomfortable position of selling our work to other divisions of Microsoft to show short-term value.
Ethnographies That Fit the Mold
Conversely, some areas where we have met great success have been projects that clearly fit within the phases of product development and the legacy model of impact: 1) providing a strong, clear connection to defined immediate or medium-term business goals; 2) providing feature-specific data from the field; and 3) evaluating products mid-stream and end-stream in the product cycle.
Fit into Near Term – A place that ethnographic work has been widely adopted and successful is in bringing deeper insight to customers and markets that have been defined as immediate or medium-term business targets. It is increasingly common for companies to invest in quantitative segmentations of their target customer base. These segmentations are expensive, and therefore senior leadership are invested in utilizing the data. However, the data is often overwhelming, making it difficult for people to use the information to help them focus. For us, ethnographic methods help to focus the quantitative data in many ways: understanding which of the statistics is the most relevant to differentiate segments, adding richness to help understand who these people really are and making the data digestible for engineers. Without a qualitative compliment, putting a face on a segment is a guessing game based on broad demographic breakouts. And more importantly, it is the story of a person that our colleagues remember – Eva hits snooze every morning when her mobile phone’s alarm goes off – not that 19% of people use the alarm clock on their mobile phones every day. Our most used ethnographic data is immediately relevant to our colleagues and comes alive in their mind’s eye in a way that statistical data does not.
However, the insights themselves have not been enough to ensure the ethnographic data will be adopted. Two pitfalls that have been experienced at Microsoft are having data intended to educate a product team on their target customer collected and released after the product cycle had already begun, thus missing the planning phase where team members are creating their vision of who they are designing for, and data being packaged in ways that were not memorable or consumable enough for product team members. Therefore to ensure success we have learned that it is necessary to collect our data during the end of one product cycle and have our insights and deliverables ready for consumption the day the new product cycle launches. Additionally, we have learned that our design colleagues are essential in helping us to create various ways to communicate the information that resonates with our internal audience.
Feature-Specific Data – Another way that ethnographic methods have been successful within the existing model of impact is to collect feature-specific data while in the field and provide that information to usability colleagues. This type of data can be collected within the scope of almost any project and does not need to be the focal point of the research. For example, within the course of an ethnographic deep dive with one customer segment, we are sure to include questions that our usability engineers have provided us such as ‘how do you add a new contact to your mobile phone?’. This type of data may not naturally emerge during observation, but it is easy enough to have a set of feature-specific questions that we ask at a closing interview with our participants. We are still able to conduct broad investigation, while providing granular information that will assist our usability colleagues, add to the information about people’s behaviors overall and molds to the existing model of impact. That being said, we are still learning how to most effectively package the raw data so that researchers can easily consume, utilize and repurpose the information.
Product Evaluation – Finally, the use of ethnographic methods to observe a soon-to-be-released product in the field to evaluate short-term improvements and gather requirements for the next version has been fully embraced at Microsoft. This has been successful because it has allowed a view into how people used the product ‘in the wild’ and the difficulties they encountered, which were often different than difficulties identified in the lab or by the testers. In addition, programs like these have made our customers really tangible for the development team and people delight in following along with the excitement and disappointment our products evoked. While there is tension that by the time we have a beta product it is at a point in the product cycle we may not have enough time to fix problems that are discovered before the product ships, the power of real people suffering through bad experiences puts tremendous weight behind these findings to be priorities. Evaluative work is an excellent opportunity to marry usability and field methods, allowing a close partnership with usability colleagues. The resulting data reveals short-term items to be fixed (adhering to usability metrics) and uncovers long term needs and goals of target users to help set up the next release.
In our tenure as practitioners at Microsoft we have been able to identify how ethnographic work can have the greatest impact. One of our biggest learnings has been that if we focus our efforts on single ethnographies that stand alone like islands, outside of the current foci of our product team, we cannot enjoy long term success. By evaluating our experiences and approaches over the years, we have learned how to get smarter about translating our findings to an engineering culture and situating some of our work within an existing impact model. Once we had gained these insights we realized we needed to find ways to promote our strengths and deep insights within the existing product cycle and engineering rhythm. We needed a new model of impact – and we needed to be able to sell it internally in such a way that its value would be widely recognized.
MAKING SENSE OF OUR STRENGTHS: ETHNOGRAPHY AND THE PRODUCT CYCLE
“Ethnography provides insight into the organization of social settings… it provides models for thinking about those settings and the work that goes on there. The value of ethnography, then, is in the models it provides and the ways of thinking that it supports.” (Dourish, 2006)
We are trained to look across fields of complexity and make sense of them for our audience. Midway through our journey of failures and successes across Microsoft, we began to more effectively apply our analytic lens on ourselves in order to identify opportunities for increasing our own impact. We began looking inwardly to the webs of business influence in which we were caught as much as we looked outwardly to the webs of customer complexity. We talked less about our methodology, and more about business goals. We branded ourselves based on our ability to decipher our target markets and deliver relevant insights, but we grew our brand by deciphering and strategically navigating our politicized organizations. We don’t think any of this is unique within our community of praxis – we know many of you have done and are doing the same.
In our context of praxis, our approach to this has been oriented around the common conceptual model of the product development cycle combined with an honest analysis of models of power and influence within the corridors of Microsoft product teams (Lovejoy & Steele, 2008; Flynn, in press). Despite the bruises we earned in making ethnography relevant, we also recognized that we were uniquely positioned as research professionals within the corporate organizational landscape. Due to its strong engineering culture, the product teams, within which traditional engineering sits, are the loci of decision-making around product and business strategy and the primary axes of influence. Although ethnographic practice could live in our marketing teams or in Microsoft Research, an exclusive engineering research & development group looking 5+ years into the future, neither of these places offer the opportunity to have extensive product impact. Market researchers are viewed as providing a service function for delivering data to marketing, not product teams, and Microsoft Research does not have strong ties to product decision-making. By sitting within the product team, we believed we were appropriately positioned to carve out a niche of influence.
Earlier we discussed ways in which ethnographic research can maximize its success within the expected discourses of research and product development inside our company. Driven by our deeper understanding of these successes and the desire to do more strategic and insightful work that would have direct product impact, we decided we needed to carve a new type of discourse into the organization. Our challenge was to introduce a holistic research lens, encompassing, but not limited to ethnographic inquiry, into a research landscape that was dually defined by usability on one side and market research on the other – and to introduce it in such a way to gain credibility of both colleagues and leadership.
We recognized the need to closely tie our value to the dominant paradigm of the product cycle. The research that our marketing teams executed tended to be front-loaded – mapping out broad landscapes of markets, segmentations, and competitors to support early product planning. Conversely, our usability colleagues maintained a narrow focus on product features, iterating and validating towards the middle and end of the product cycle. We saw a gap and an opportunity – to train our lens on the people who are embedded within and yet obliquely obscured by portraits of broad market landscapes or a narrow focus on features. While the product team and our usability colleagues were heads-down in the lab deep into the product cycle, we could be out in the world building knowledge to guide the team through the front edge of the next product cycle – and partnering with our market research colleagues to bring some design specificity to the broad opportunities that they were identifying. We launched our own Strategy Team within the User Experience discipline, framed our mandate around building knowledge of target customers and product opportunities 12 – 36 months ahead of the core product team, and translating market data into the language of users, experiences, scenarios, tasks, and features as spoken by the product team and executives (Diagram 1.2). Our UX Strategy team consisted of two ethnographers (the authors) and one usability engineer – the ethnographers driving the field research and the usability engineer creating tools and processes to appropriately insert ethnographic insights throughout the product cycle.
We found the sweet spot for ethnographic impact within our world. We produced insights that simultaneously had direct product impact to our engineering partners and that pushed the boundaries of our corporate assumptions. Through careful planning, strategic project prioritization, deliberate partnership development and our craft of developing and delivering customer, product, and business insights with a strong dose of storytelling, we soon had the ears of the top executives for the product division. We waved flags of missed opportunities and rich target markets that our business couldn’t ignore, and enjoyed repeated success in pushing the front-edge of critical waves of change that gained divisional momentum and transformed our business priorities. We became the go-to team for customer-focused decisions, discussions, or questions for teams and executives. We were also able to take ethnography to new places inside Microsoft – moving beyond the role of knowledge expert and opening the door to the boardroom to take a seat beside our engineering counterparts in making decisions about our product and business strategy.
More recently a number of developments have come together to help us further evolve and move towards a more integrated product research discipline with our usability partners. As the first ‘ethnographer’ to lead and manage an entire user research team at Microsoft, and with executive support to create a more holistic practice, Donna has been empowered to drive a vision of end-to-end research across a product cycle. Strategic researchers and usability researchers are identifying new ways to stretch themselves from the front-end of design planning to the tail-end of product testing without losing the value of their specific lenses, and to further interweave their approaches to bring value across the entire product cycle.
MAPPING OUR SENSE-MAKING JOURNEY: A TAXONOMY OF ETHNOGRAPHIC IMPACT
In this journey of making sense of our strengths within our context of praxis, we were able to break through a new ceiling of influence as ethnographers within Microsoft. We delivered clear value in aligning customer understanding with product strategy, created a strong brand for our team, and became widely recognized as thought leaders in user experience. We no longer felt we were swimming against a strong tide in convincing colleagues about the value of our approach, or even about how it fits into our processes. In referencing our work in an internal video, our Corporate Vice President claimed “Never before have we understood our market data as deeply as we do now.”
These achievements also helped define a new, accepted model of impact for product research practices inside our organization. In addition to the legacy usability paradigm of impact, an ethnographic paradigm of impact has been established in its own right (Diagram 1.3).
Through this exercise of mapping our own sense-making journey, we have identified clear stages in an arc of ethnographic impact within our organization (Diagram 1.4). Climbing this arc has been a process of gaining visibility and credibility across our teams – both for ethnography as a practice and for us as individuals. For the arc as individual growth, what do we need to do to more efficiently move practitioners up the trajectory? From an organizational perspective, a mature practice will have individuals embedded across all of these stages in the organization, providing both entries and growth paths for the next generation of leadership.
For the arc as practice growth, finding the appropriate way to engage in the dominant paradigms and redefining expectations of value were necessary triggers for moving to a new level of increased influence. The potential for ethnographic authority was established by changing the order of things. At Microsoft, this has taken a number of years and there is also a vast organizational horizon – even though we have redefined models of impact for our own division of 2500+ people, how do we effectively scale or extend across the company of 75,000 people? Will it be easier for the next team who attempts the same, or do we find ourselves continually starting at the bottom of the arc in our quest for ethnographic authority? Also, our arc of impact is situated within our business context. How do the arcs differ for those of you in consultancies or government organizations? Our biggest challenge as practitioners is to redefine perceived value beyond our immediate contexts of praxis – across our corporate terrains and across the business landscape—and collectively move towards a replicable model that will help us define our lasting place in the order of things.
MORE SENSE TO MAKE: FURTHER QUESTIONS
We want to end this conversation with more questions than answers. As we climb the arc of impact, how does our identity change – as individuals and as a practice? Our individual public identities have evolved as our roles have grown. One outcome of our success at growing organizational integration, executive mandate, and strategic influence of ethnography is increasing recognition as leaders and managers. In turn, we find ourselves leading researchers (and designers) of all stripes, have moved further away from our everyday identity as ‘ethnographers,’ and reduced the frequency of our research cycles.
In many ways, our ethnographic lens has provided a natural path towards leadership – enabling our strategic perspectives and ability to see the big picture complexity of both customers and our organization. But does our identity as ethnographers also pose a glass ceiling in our organizational context, and constrain our ability to move into higher leadership positions by compartmentalizing our capabilities in the eyes of executives?
We are also now seeing a wave of ‘holism’ slowly sweep across other teams and practices within Microsoft, with usability researchers claiming the ‘field’ as one of their arenas of research. Is our relative value thus more about building a vision for end-to-end strategic research that is often born from an ethnographer’s holistic perspective and training? Will these trends threaten our authority over our historical domain, or are we witnessing a trend that will embed ethnography even more fully into hegemonic structures?
Donna K. Flynn leads the User Experience Strategy & Research team for Microsoft’s Mobile Communications Business. Over the past 12 years she has led strategic research on a wide range of business and consumer users, specializing in international ethnographic approaches. Since receiving her PhD in Anthropology from Northwestern University in 1997, Donna has worked across diverse applied contexts, including international development, business consulting, technology and product design.
Tracey Lovejoy is an ethnographer and user experience lead who has worked at Microsoft since 2001. Her projects have been wide in range, from baby boomers to teens to enterprise organizations across North America, Europe, Asia and South America. Tracey is the co-founder of the Ethnographic Praxis in Industry Conference, which brings together people who are actively thinking about the theoretical and methodological development of ethnography in industry. She received her B.A. in International Relations from UCLA and her M.A. in Social Science from the University of Chicago.
Ethnographic Praxis in Industry Conference Proceedings 2008, pp. 238–250. © American Anthropological Association, some rights reserved.
REFERENCES
Beyer, H., & Holtzblatt, K.
1998 Contextual design: Defining customer-centered systems. San Francisco, CA: Morgan Kaufmann Publishers.
Blomberg, J., Burrell, M. & Guest, G
2003 An ethnographic approach to design. The Human-Computer Interaction Handbook, eds Jacko, J.L. & Sears, A., 964-985. Lawrence Erlbaum Associates: New Jersey.
Dillon, Andrew
2000 Group dynamics meet cognition: combining socio-technical concepts and usability engineering in the design of information systems, in: E. Coakes,. (et al) (Eds.) The New Socio Tech: Graffitti on the Long Wall. London: Springer-Verlag, 2000, pp119-126.
Dourish, Paul
2006 Implications for Design. Computer Human Interaction, Montreal, Quebec, Canada.
Dumas, Joe
2007 The great leap forward: The birth of the usability profession (1988-1993), The Journal of Usability Studies, Vol. 2, Issue 2, February 2007, pp. 54-60.
Flynn, Donna
In Press My Customers are Different!: Identity, Difference, and the Political Economy of Design. In Ethnography and the Corporate Encounter: Reflections on Research In and Of Corporations. M. Cefkin, ed. New York, Oxford: Berghahn Books.
Lovejoy, Tracey and Nelle Steele
2008 Effectiveness through Partnerships. In Partnering for Organizational Performance, Elizabeth K. Briody and Robert T. Trotter, II, eds., Lanham, MD: Rowman and Littlefield Publishers, Inc.
Nielsen, Jakob
1993 Usability Engineering. Academic Press, Boston.
Roberts, Simon
2006 The Pure and the Impure? Reflections on Applying Anthropology and Doing Ethnography. In Applications in Anthropology: Professional Anthropology in the Twenty-first Century. S. Pink, ed. Pp. 72-89. Studies in Applied Anthropology. New York, Oxford: Berghahn Books.
Robinson, Rick E.
1994 Making Sense of Making Sense; Frameworks and Organizational Perception. Design Management Journal, Winter 1994.
Sherry, John
2002 Foreword: Ethnography, Design, and Customer Experience: An Anthropologist’s Sense of It All, Squires, Sue and Bryan Byrne, eds., Creating breakthrough ideas. Westport: Bergin Garvey.
Sunderland, Patricia L and Rita M Denny
2007 Doing Anthropology in Consumer Research, Left Coast Press