Why Becoming a Data-Driven Organization Is So Hard

 Companies have been working to become more data-driven for many years at this point, with mixed results. These efforts play out over time in organizations, and persistence, resilience, execution, and a relentless drive to employ data to make more informed business decisions are what distinguishes those companies that prevail from those who continue to struggle. But while the mission may remain steady, the particulars change.

Right now, the biggest challenge for organizations working on their data strategy might not have to do with technology at all. In the latest NewVantage Partners annual survey, which tracks the progress of corporate data initiatives, corporate chief data, information, and analytics executives reported that cultural change is the most critical business imperative. It’s an understandable problem: to a degree that is perpetually underestimated, becoming data-driven is about the ability of people and organizations to adapt to change. Long-established companies, which have been successful over generations or centuries, are unlikely to change overnight — adoption of the internet through digital transformation efforts has played out over the course of the past quarter century. Similarly, the effort by companies to become data-driven represents a business transformation that is playing out over the course of a generation. Much has been achieved, yet more remains to be done.

But while this issue isn’t new, there are two cultural dynamics that have shaped company efforts during the past few years.

First, the Covid-19 pandemic — and the disruptions it caused — raised awareness of the importance of data, science, and facts. While companies may have paid lip service to the importance of data before, the case that good data is essential to making informed, prudent, and judicious business decisions has been made crystal clear over the past two years.

Second, self-service is on the rise, and individuals now consume information and data when they want and how they want it. We live during a time of increasingly decentralized information, which means that consumers can select the news they follow, the social media they engage with, and the data that they choose to trust, with the consequence that consumers of information can be subjected to selective presentation of data to support a wide range of often diverging viewpoints. At its most extreme, this has given rise to the notion of “alternative facts.”

Finally, there’s a structural fact: the amount of data that is created each day continues to proliferate at exponential rates. With greater computing power, companies can now process massive quantities of data to generate a precise answer, rather than rely on representative data samples.

Understanding these trends — and how other companies are navigating them — can help companies make real progress towards their goals of data-driven decision making.

Barriers to Becoming Data-Driven

There are three indicators of progress that stand out among the surveyed organizations. First, achieving data-driven leadership remains an aspiration for most organizations — just 26.5% of organizations report having established a data-driven organization. Second, becoming data-driven requires an organizational focus on cultural change. In this year’s survey, 91.9% of executives cite cultural obstacles as the greatest barrier to becoming data driven. As noted, this is not a technology issue. It is a people challenge. Lastly, organizations are establishing the leadership function — in the role of the Chief Data and Analytics Officer — which will provide the foundation for becoming data-driven. However, just 40.2% of companies report that the role is successful and well established within their organization.

It doesn’t help that the task of being data-driven keeps getting harder. Today, corporations encounter vast new volumes of data, as well as new sources of data, which include sensor data, signals, texts, pictures, and other forms of unstructured data. It has recently been argued that 80% of all new data is unstructured, meaning that it is not easily captured or made quantifiable. Increasingly, companies must come to recognize and appreciate that data is a business asset that flows through an organization. Data cuts across traditional organizational boundaries, often without clear ownership. The fluidity of data compounds the complexity of managing this asset in a way that consistently delivers business value.

Furthermore, there is one rapidly emerging concern that confronts every business these days when it comes to the ownership and management of data. That is the assurance of responsible and ethical data use. This is a topic that has been written about extensively in recent years and has been the subject of critics ranging from Cathy O’Neill, in her 2016 manifesto Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy to Shoshana Zuboff, in her 2019 call to arms, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. Recently published works by Carissa Veliz, Privacy is Power: How and Why You Should Take Back Control of Your Data (2021), and Why Privacy Matters (2021) by law professor Neil Richards, dig deeper into the issues of individual privacy and corporate data responsibility.

This year’s survey mirrors and highlights the depth of corporate concern regarding data ethics and data responsibility, reflected in the meager 21.6% of data leaders who state that the industry has done enough to address data and AI ethics issues and standards.

Steps Companies Can Take

Becoming a data-driven organization is a journey, which unfolds over time, measured in years, and sometimes decades. What steps can organizations and business leaders take to accelerate these efforts? Experience tells us that data-driven organizations consistently demonstrate qualities that distinguish them from their contemporaries. Data-driven companies consistently execute on these three driving principles:

1. Think different. Data leaders recognize that becoming data-driven requires a different mindset. Organizations must be prepared to think differently. There is no shortage of analytic algorithms. These need to be matched by critical thinking, human judgement, and a view to creative innovation.
2. Fail fast, learn faster. Data leaders understand that individuals and organizations learn through experience, which often entails trial and error. It has been said that failure is a foundation of innovation. Companies that are prepared for faster iterative learning — fail fast, learn faster — will gain insight and knowledge before their competitors.
3. Focus on the long-term. Data leaders appreciate that the data journey is a transformation effort that unfolds over time. Becoming data-driven is a process. The French writer Voltaire famously said, “Perfect is the enemy of good.” Perfection is rarely achievable. Data-driven companies recognize that success is achieved iteratively. It will grow and then spread. Successful organizations expect to be at this for a while. They focus on the long-term.

To compete in the increasingly data-driven world of the twenty-first century, business leaders must learn from the experience of their predecessors. They must actively work to avoid the pitfalls of the past, and benefit from the example of those companies that have pushed forward with success. Now more than ever, at a moment when data, science, and facts have been challenged from many quarters, becoming a data-driven organization matters.