Overview
In this episode of The Dashboard Effect, Brick Thompson and Caleb Oaks take stock of where generative AI in data analytics actually stands, measured against the expectations that were set when the technology arrived with considerable fanfare at Microsoft Build in 2023. The assessment is candid and grounded in real experience: the tools have improved, but the gap between what was promised and what reliably works in production remains significant.
For organizations that have been waiting for generative AI to transform their analytics workflows, this episode offers an honest calibration of what is reasonable to expect right now and what the path forward might actually look like. See how Blue Margin’s Managed Data Platform builds the semantic layer, data governance, and trusted data foundation that will determine whether your organization is positioned to take advantage of AI analytics when the technology matures enough to deliver reliably on its promise.
What This Episode Covers
Initial Expectations vs. Current Reality (1:22 – 2:34)
The early excitement around tools like Power BI Copilot and ChatGPT for data analysis generated expectations that the technology has not consistently met. The hosts describe results that are often unreliable enough to require substantial manual verification before they can be trusted, which largely defeats the efficiency gains the tools were supposed to deliver. The experience feels more bleeding edge than production-ready, and they are direct about that assessment.
Data Integrity and Semantic Layers (2:34 – 6:42)
The garbage in, garbage out principle applies to generative AI as absolutely as it does to any other analytics tool. Building the semantic models required to give LLMs a meaningful context to work within is a complex, high-overhead undertaking, and even with well-constructed models in place, LLMs continue to struggle with logical reasoning and mathematical accuracy in ways that create real reliability problems for business analytics use cases.
The Trough of Disillusionment (7:29 – 8:47)
Brick and Caleb are willing to name what many in the industry are quietly experiencing but reluctant to say publicly: generative AI analytics tools are currently in the trough of disillusionment. The initial marketing set expectations that the technology is not yet consistently meeting, and organizations that invested early in these capabilities are now working through the gap between the demo and the reality.
Future Outlook (8:47 – 10:28)
Looking further ahead, the hosts raise the possibility that LLMs as currently architected may be approaching a performance ceiling for analytics use cases. The reasoning and mathematical limitations that make current tools unreliable for business analytics are not simply a matter of scale. They may require a fundamentally different approach, potentially involving specialized agent architectures or other structural changes, rather than further refinement of existing language models.
Where LLMs Are Still Useful (10:28 – 11:19)
The hosts are not dismissive of generative AI broadly. They continue to use LLMs for tasks like brainstorming, drafting, and exploration where the cost of an imprecise output is low. The distinction they draw is between those use cases and production analytics workflows where accuracy is non-negotiable. In the latter category, they do not anticipate truly reliable generative AI tools arriving in the immediate future.
Who It’s For
This episode is worth your time if you are a business or technology leader who invested in generative AI analytics tools and is trying to make sense of why the results have been inconsistent, a data team evaluating whether to build workflows around LLM-based analytics or hold off until the technology matures further, an executive trying to set realistic expectations internally about what generative AI can and cannot deliver for your analytics function today, or anyone following the evolution of AI in the analytics space and wanting a grounded perspective from practitioners who are actively working with these tools.
Why It’s Worth a Listen
The generative AI conversation in analytics has been dominated by optimism, and there is a real cost to that imbalance when organizations make investment decisions based on capabilities that do not yet exist at production scale. This episode provides a counterweight that is not pessimistic about the technology’s long-term potential but is honest about where it currently falls short and why.
The discussion of semantic layers and their limitations in the context of LLMs is particularly valuable. A common assumption is that sufficiently good data preparation will unlock reliable generative AI analytics. Brick and Caleb challenge that assumption directly, pointing to reasoning and mathematical accuracy as limitations that better data alone does not solve. That distinction matters for teams deciding where to invest their preparation efforts.
And the framing around the trough of disillusionment is useful precisely because it is borrowed from a well-understood pattern in technology adoption. Naming where the industry is in that cycle does not mean the technology will not mature. It means the organizations that survive the trough with their expectations intact will be better positioned when the tools actually deliver on what was promised.
