Benchmarks

Deeptrace-Bench Hard

An evaluation of AI agents across 91 of the hardest engineer-verified, real-world production incidents and alerts our customers have reported. On our benchmarks, Deeptrace vastly outperforms Claude Code with MCP plugins.

Overview

Deeptrace (64.8%) performs 17.5% better than Claude Code (47.3%) across 91 real-world production incidents/alerts. We sourced this data across dozens of different environments from anonymized customer data.

Deeptrace and Claude Code had access to the same APIs, integrations, and data. Both were evaluated without any human intervention, confiruration, or custom prompting.

*To evaluate Claude Code (Opus 4.6), we used MCP plugins with access to the same APIs and data sources as Deeptrace.

Overall Accuracy
100%80%60%40%20%0%
Deeptrace
Claude AIClaude Code*
Root Cause Accuracy by Severity Level
100%80%60%40%20%0%
Critical
Medium
Low
Deeptrace
Claude AIClaude Code (Opus 4.6)

Methodology/Dataset

The dataset covers 91 incidents and alerts across dozens of real production environments, spanning infrastructure, application, and data pipeline failures.

For each case, we have an engineer-verified root cause from an engineer on the team that manually debugged the issue. On each of these cases, we concurrently ran an instance of Deeptrace and Claude Code (Opus 4.6) with directions to root cause the issue.

Both Deeptrace and Claude Code had access to the same underlying data and MCP integrations. A root cause was considered “correct” if the final response matched the engineer-given root cause.

Incident types

API & webhook failures24
Infrastructure & scaling19
Data pipeline errors16
Deployment & CI/CD13
External service outages10
Business logic bugs9

Integrations used

Datadog42GitHub38Sentry27PagerDuty22CloudWatch18Grafana14PostgreSQL12Linear11incident.io9New Relic7Rollbar5GCP4

Example Cases

The following are anonymized examples of cases where Deeptrace correctly identified the root cause but Claude Code did not.