Monitoring AI workloads has made jobs more challenging: Splunk

Patrick Lin, the SVP and GM of Observability at Splunk, a Cisco company

The global Splunk State of Observability 2025 report reveals observability insights are guiding key business decisions in customer experience, product roadmap forecasting, and brand perception. That’s among the results of the report.

“Observability practitioners are becoming critical stakeholders to key business decisions in customer engagement strategies, product roadmaps and more,” said Patrick Lin, the SVP and GM of Observability at Splunk, a Cisco company. “And this year’s State of Observability report findings make that clear: the full life cycle and workflow of observability – from data collection and analysis to deriving actionable insights and implementing improvements – provides not just better context, but also support for the achievement of better results, whether in customer satisfaction, product innovation or the safeguarding of AI systems at scale.”

“Observability leaders generate an annual 125% ROI from their observability practice — 53% higher than their peers.,” Lin said. In addition, the report’s findings show that observability boosts employee productivity for nearly three‑quarters of respondents, and for nearly two‑thirds, it drives revenue growth and helps shape product roadmaps. In addition, nearly half of those surveyed say monitoring AI workloads has made their jobs more challenging, presenting an opportunity to train practitioners on essential expertise. Observability leaders also differentiate themselves by adopting innovative practices and frameworks – such as OpenTelemetry, code profiling, and observability-as-code. It’s a far cry from observability’s origins, when teams worked quietly behind the scenes to keep the lights on by maintaining the business’s applications and infrastructure, and finding and fixing issues. Now, observability practitioners aren’t only supporting the business; they’re acting as catalysts to drive the business forward.

“Observability practitioners are becoming critical stakeholders to key business decisions in customer engagement strategies, product roadmaps and more,” Lin said. “And this year’s State of Observability report findings make that clear: the full life cycle and workflow of observability – from data collection and analysis to deriving actionable insights and implementing improvements – provides not just better context, but also support for the achievement of better results, whether in customer satisfaction, product innovation or the safeguarding of AI systems at scale.”

To explore how observability is evolving in this high-stakes environment, Splunk surveyed 1,855 ITOps and engineering professionals across nine countries and 15 industries for their latest report, State of Observability 2025: The Rise of a New Business Catalyst. The research reveals how the highest-performing teams are driving outcomes that impact the broader business. They’re nearly twice as likely as their peers to say that their observability practice significantly improves overall revenue, employee productivity, and product roadmaps.

More than ever, organizations recognize that software decisions ripple across customer experience and brand reputation. As a result, observability data is shifting from a troubleshooting tool to a strategic asset, answering questions like “What can we do better?” rather than “What’s broken?” As such, organizations are prioritizing the ability to monitor critical business processes through their observability software; 74% rate it as moderately to very important.

These shifting priorities have paid off. Lin said that 65% of respondents say their observability practice positively impacts revenue, and 64% say their observability practice positively impacts product roadmaps.

Splunk’s research found that a distinct group of leaders influenced these business outcomes more heavily than the rest, and they also generated an ROI that was 53% higher than their peers. What they had in common was a top-tier tech foundation; they often or always used OpenTelemetry, code profiling, and observability-as-code. They also collaborated more closely with their security counterparts. 44% strongly agreed that ITOps and engineering teams troubleshoot and solve issues with security, compared to 29% of others, and used emerging AI technologies like agentic AI at higher rates than their peers.

That’s the good news. The bad news is that many teams still struggle to shake off their reactive roots. A full 43% of respondents admit they spend too much time responding to alerts. And 20% say they often or always start a war room with members of many teams until an issue is resolved, underscoring how common, and costly, firefighting remains. Alert fatigue is taking its toll, as 73% report experiencing outages due to ignored or suppressed alerts. Prioritizing alert hygiene and incident response is crucial because alerts are so tightly woven to business success; 54% say the quality of their alert detections has the greatest effect on observability ROI.

One route to smoother incident resolution is collaborating with security teams; Lin reported that 54% say that it wastes less time chasing down issues. And the benefits extend to the broader business, too; 64% say it leads to fewer incidents that affect customers.

“For a modern business, built on digital experiences, observability is not just about error resolution; it is a foundational discipline required for making business-shaping decisions at speed and scale,” said Shannon Kalvar, Research Director at IDC.

The global survey was conducted from February through March 2025 and surveyed 1,855 ITOps and engineering professionals from practitioners to VP-level executives (including developers, SREs, systems engineers, infrastructure operations professionals, CTOs, and CIOs.) The survey respondents were drawn from nine countries: Australia, France, Germany, India, Japan, New Zealand, Singapore, the United Kingdom, and the United States. Respondents also represented 16 industries: business services, construction and engineering, consumer packaged goods, education, financial services, government (federal/national, state, and local), healthcare, life sciences, manufacturing, technology, media, oil/gas, retail/wholesale, telecom, transportation/logistics, and utilities.