How to find signal in the noise – managing large amounts of data sources and finding meaningful insights

At Encompass, data underpins everything we do. Data is essential for making informed decisions, driving the right transformation outcomes and ultimately improving business performance. 

Every millisecond, a tremendous volume of data is generated in the age of the internet and mobile devices. We are operating in a data rich landscape with collection now being possible across multiple touchpoints. Although this provides many opportunities, it also brings challenges for organisations to find the right business value, aka insights amongst all the discrete pieces of data. Even with excellent quality data and advanced technologies, gaining insights can still be difficult.

As the famous statistician, Nate Silver noted

 “The signal is the truth. The noise is what distracts us from the truth.” 

So how do we best find this invaluable gold of truth and avoid getting lost in a world of distractions?

From Encompass’ experience, finding the truth requires the right combination of technology, data skillset and understanding of the relevant business context. Without the combination of these factors, the output runs the risk of becoming yet another business report or dashboard that makes its way to an Executive’s top drawer.

Before commencing any form of analysis, it is important to have clarity on the business problem you are trying to solve. This then dictates the subsequent data selection, blending, interrogation and final presentation phases. 

Our top tips for finding the signal (insights) in the noise of data are highlighted below.

1. Define:  Before starting any form of analysis it is important to think about the business problem you are trying to solve. What are the key performance metrics that are currently available and may be influential? If lead indicators exist, these can often be a good starting point. Many data deep dives fall apart as there is misalignment on this first step. 

2. Data Selection: A good data set is the foundation of any analysis, so gathering the right data is critical. This can be as simple as extracting it from an existing data warehouse or as complex as generating a feed from a web stream or even creating a new data source. At Encompass we have experience leveraging multiple technologies to extract the data whether it be a simple query or creating a whole new process of data integration. 

3. Data Blending: To harness the power of data, it is always suggested that you bring all the pieces of information together and then apply the business lens. Data blending is the process of stitching together data sitting in silos to achieve the complete data set required for the analysis. A single data source is never enough to investigate and generate insights. For example, you may have separate data sets of different areas of a process, to get a complete view of the end-to-end process you need to connect all these sets together in a consistent way. Depending on the complexity of the available data, the Encompass data team leverages tools such as Alteryx, Microsoft Excel, IBM Data Stage and Informatica to successfully complete this step.

4. Data Interrogation: A thorough analysis of the data set is performed to determine what is significant in relation to the problem statement/research objectives. By asking the right questions it allows you to slice and dice the datasets to identify any interesting and notable themes, patterns, trends, and issues. Encompass uses tools such as Excel, Tableau, Power BI, Qlik, to drive exploration via graphs and matrixes that enable simple and flexible ways to examine different data patterns. The outcome of the process is an in-depth, explorative study of the problem statement and any key contributing factors, common themes, or unique outliers. This is a collaborative process which involves consultation with business SMEs and data teams to align on business rules and whether the themes and patterns generated are valid. Remember, there is no easy way to shortcut this process and unfortunately finding valuable insights often takes time! 

5. Present: This can often be the key to success of the entire process as it is the final product that allows business stakeholders to visualise some of the trends and ultimately enable effective decision making. It is important to aggregate the presentation to the right level of detail and remember stakeholders have not been living and breathing this data so will require some form of additional context.

Encompass tends to follow a consistent structure for their deep dive analysis presentations:

  • Clear articulation and definition of the scope and the problem statement
  • Purpose and desired outcome of the analysis
  • Key assumptions and callouts for data 
  • Trajectory graphs to show trending towards key targets in ‘change nothing’, ‘lower range’ and ‘upper range’ scenarios (see our Building meaningful trajectories article)
  • Visuals and charts with a Red, Amber, Green (RAG) status to provide clear and focussed information
  • Callouts, key insights and overlay of inflight initiatives to determine contribution 
  • Conclusion and next steps. It’s important to agree some clear actions or decisions off the back of all insights. 

Are you ready for transformation?

Encompass Consulting Services partners with state government, federal government and corporates to develop tailored solutions that turn strategy into results. Driven by evidence-based decisions, data underpins everything we do. We leverage metrics and analytics to identify gaps, inform priorities and drive delivery. Get in touch to improve your organisation’s data fluency, measure performance, and shift from subjective to objective decision-making. 

Looking for support? Reach out to our data and analytics Senior Manager, Rahul Singh, rahulsingh@encompassconsultingservices.com

report

How to Find Signal in the Noise

Download report

Categories

Popular posts