Automating Root Cause Analysis for CI/CD Failures Using AI at Halodoc

Introduction

CI/CD pipelines are an essential part of modern software delivery, helping engineering teams build, test, and deploy applications continuously. At Halodoc, our CI/CD environment runs roughly 800 builds every day, with around 190 ending in failure. At this scale, manually investigating failures can create significant engineering overhead, especially when the root cause is buried within thousands of lines of Jenkins logs.

To reduce this investigation effort, we built an automated Root Cause Analysis (RCA) Agent as part of our AI-Powered Pipeline Intelligence initiative. The agent is triggered directly from a post-build step in the shared Jenkins pipeline library — not a separate watcher or poller — so it fires the moment a build (or, for parallel stages, an individual branch) is marked FAILURE, UNSTABLE, or ABORTED. From there, it collects the relevant failure evidence, identifies the likely root cause, and posts a concise analysis directly to the engineering team's Google Chat channel.

The agent is designed to handle failures using the simplest reliable approach: known failures are identified through deterministic rules, recurring failures can reuse previously validated diagnoses, and unfamiliar failures are analyzed using AI. For specific infrastructure failures, the agent can also perform controlled retries before escalating the issue to the SRE team.

Why We Built This

The need for automated RCA became clear from two recurring challenges in our pipelines.

First, infrastructure failures could look like application failures. An agent being OOM-killed or evicted midway through a build could leave errors that appeared similar to compilation or test failures. Engineers often had to investigate deeper into the logs to determine whether the problem was in the application or the underlying infrastructure.

Second, several of our pipelines run validation stages in parallel. Jenkins fail-fast behavior can interrupt sibling stages when one stage fails, meaning a stage may appear unsuccessful even though it was not the original source of the failure.

These challenges showed that a failed build does not necessarily explain why it failed. The system needed to identify the actual failure, separate the root cause from secondary errors, and determine what action should be taken next.

Choosing the Right Approach

The obvious approach was to send every failure to an LLM. We deliberately didn't do that.

Many of our failures are predictable. If a coverage report is missing or a known infrastructure issue occurs, there is little value in spending time and resources asking an AI model to identify something that can be determined reliably through a rule.

Instead, we wanted the system to use the simplest approach that could reliably handle each failure:

• Known failure → rule
• Recurring, previously analyzed failure → reuse the diagnosis
• Unknown failure → AI analysis

This became the foundation of our three-tier RCA approach, allowing us to reserve AI analysis for failures that genuinely require deeper reasoning.

Architecture

The RCA Agent follows a simple decision flow: a Jenkins build fails → the pipeline's post-build step immediately invokes the agent (no polling, no delay) → it collects the relevant failure evidence → classifies the failure → analyses it when necessary → takes the appropriate action → notifies the team in Google Chat

When a build fails, the agent first fetches the Jenkins console log and identifies the relevant failure evidence. It then checks whether the failure matches a known pattern. Known failures are handled immediately by Tier 1 rules. For recurring failures, the agent uses the failure signatures and diagnosis rules being collected from production data. Failures that cannot be classified through these deterministic paths move to the AI analysis tier.

The final alert includes the Issue Type, relevant evidence, root-cause analysis, and suggested fix. For code-related failures, the recommendation remains advisory and the engineer decides what to do next.

Infrastructure failures follow a separate, narrower recovery path. When the agent detects failures such as an agent being killed, OOM, or eviction, it retries the build using the same parameters as the original run.

If the retry succeeds, the failure is resolved. If the same job continues to experience consecutive infrastructure failures and reaches the configured threshold, the agent stops retrying and escalates the issue to the SRE team with an @mention.

Example Flow — A Real Coverage-Regression Alert

To make the decision flow concrete, here's a real example from production: an iOS build that regressed below its coverage gate.

In this case, the failure matched a known coverage pattern, so the agent handled it through Tier 1 without making an AI call. It extracted the affected module from the failure evidence and included the specific reason and suggested fix in the Google Chat alert.

Example Flow — An AI-Analyzed Failure

To show the other side of the decision flow, here's a real alert where the jenkins-failure-analyzer skill was consulted for analysis.

In this case, the failure — a Gradle OOM during a Robolectric test run — matched a recognized failure pattern, but pinpointing the specific fix still required deeper reasoning. Rather than returning a generic recommendation, the agent consulted the jenkins-failure-analyzer skill once to analyse the stack trace and generate a fix suggestion. The suggested fix — checking JVM heap settings in build.gradle tied to a Robolectric memory issue — was posted directly to the Google Chat alert for the engineer to verify.

Core System Components

The RCA Agent is built around a few focused components, each responsible for a different part of the failure-analysis flow. The important design choice is that not every failure needs AI. We use deterministic logic wherever the existing pipeline evidence is enough and reserve AI analysis for failures that require deeper reasoning.

Tier 1: Rules Engine

The first layer handles known and deterministic failure patterns. It matches the failure reason and available evidence against rules built from failure patterns we have encountered in our pipelines.

We currently handle patterns such as:

• Coverage regressions
• Dependency resolution failures
• Deployment-target mismatches
• Lint violations
• Other known pipeline and build failures

When a failure matches a rule, the agent classifies it and generates a specific recommendation without making an AI call.

The rules are implemented in the RCA Wrapper Agent:

RCA Wrapper Agent — Rules Engine

The pattern is straightforward: match the failure → extract useful context → generate a specific fix.

For example, a coverage failure may already contain the affected module in the report path. Instead of returning a generic message such as "increase code coverage," the wrapper extracts the module name and includes it in the recommendation.

This is deliberately deterministic. When the pipeline already provides enough evidence to identify the problem, there is little value in sending it to an AI model.

Tier 2: Signature Cache

The second layer focuses on identifying recurring failures that have already been analysed.

The agent is currently collecting failure signatures and corresponding diagnosis rules from production data, stored in a MySQL database. These signatures capture recurring failure patterns and the diagnosis associated with them.

Once this data has been collected and validated, the agent will be able to reuse previously confirmed diagnoses for recurring failures instead of sending the same failure to the AI analyser again. In practice, this means a failure that initially required Tier 3 AI analysis can, once its signature and diagnosis are validated, be resolved by Tier 2 on subsequent occurrences — reducing both analysis latency and unnecessary AI usage over time. This is expected to reduce analysis latency and unnecessary AI usage, while continuously improving the knowledge available to the RCA system.

Tier 3: AI Analyser

Failures that cannot be classified by the rules engine move to the AI layer.

Instead of sending a raw Jenkins console log to Claude with a generic instruction to 'find the root cause,' we built a purpose-specific jenkins-failure-analyzer skill and load it into Claude Code. The skill provides structured instructions and domain-specific knowledge about our Jenkins pipelines, guiding Claude through a consistent failure-investigation process

Before logs reach the analyzer, Jenkins' credential-handling mechanisms mask tokens and secrets in the console output, so the model does not receive raw credentials. However, this masking can also remove useful diagnostic evidence that the agent may need to identify the root cause.

The skill understands conventions used by our shared Jenkins pipeline library, including:

• The color-coded logging scheme used by our pipeline stages
• catchError-wrapped policy gates, where a real failure can occur while the pipeline continues
• Different types of errors that can appear in the same console output
• The order in which an engineer familiar with these pipelines would investigate the log

The analyzer prioritizes evidence in roughly this order:

1. Errors from the shared pipeline library
2. Explicit policy or quality-gate failures
3. Native Jenkins or build-step exceptions
4. Infrastructure and network errors

This context is important. The model is not simply being asked to understand a generic CI/CD failure. The jenkins-failure-analyzer skill gives Claude a repeatable, pipeline-aware investigation process, helping it distinguish the actual root cause from secondary errors and noise in the console output.

The analyser then returns the likely root cause, supporting evidence, and a suggested fix when one can be determined from the available information.

Over roughly 10 days of observation, the AI analyser processed approximately 120 Tier 3 failures. The suggested root cause fully matched the engineer's independent finding in roughly 65% of cases, and was partially correct — identifying the right failure category but missing some specific detail — in another 20%. This is an early sample rather than a long-term measurement, and we're continuing to track accuracy as usage grows.

Issue Type Classification

Before the alert is sent, every failure is classified as either:

• Code Issue — application, test, dependency, or configuration-related failure that requires engineer review.
• Infra Issue — infrastructure-related failure that may qualify for automated recovery.

This classification gives engineers immediate context in the Chat alert and determines whether the failure is eligible for the automated retry and escalation flow.

Retry and Escalation Controller

Infrastructure failures follow a separate recovery path because some of them can be safely retried.

When the agent identifies an infrastructure failure such as an agent being killed, OOM, or eviction, it retries the build using the exact parameters from the original build. This ensures that the retry reproduces the original execution conditions rather than introducing additional variables.

Other infrastructure issues are handled differently. When the failure originates from the shared Jenkins pipeline library, the agent uses an AI skill to generate a fix and raises it as a merge request, tagging the respective SRE team for review. VPN-related pod termination issues are sent directly as an alert to the IT team, since they require infrastructure-side intervention rather than a retry or a code fix.

The controller tracks consecutive infrastructure failures for the same job. If the retry succeeds, the recovery process stops. If the job continues to experience infrastructure failures and reaches the configured threshold of three consecutive failures, the agent stops retrying and escalates the issue to the SRE team with the relevant context and an @mention.

The retry mechanism is intentionally bounded. The goal is not to let an AI system repeatedly restart builds, but to provide a controlled recovery path for a well-defined class of infrastructure failures.

Key Design Decisions & Trade-offs

The classification logic turned out to be the easy part. The harder challenge was making sure the evidence reaching the agent was accurate, complete, and representative of what actually happened in the pipeline.

During the rollout, we discovered several issues that had nothing to do with the model's ability to reason — they were problems in the evidence and integration layer, which in production can be just as important as model accuracy.

1. Evidence Completeness and Accuracy

The console log available to the agent isn't always identical to what an engineer sees during manual investigation. Jenkins' own credential-handling mechanisms, for example, once masked part of the dependency information in a real failure before it reached the analyser — the agent could still identify the failure category, but the incident exposed a fundamental limitation: the model cannot reason about evidence it never receives. This shifted our focus from prompt tuning to improving the completeness of the evidence pipeline itself.

Build status compounds this problem. Our pipelines run stages in parallel, and Jenkins can interrupt sibling stages when one fails — so a stage can appear failed or incomplete without being the original source of the problem. The agent therefore prioritizes the actual failure evidence and execution context over the raw Jenkins status when determining the root cause.

2. Platform and Context Awareness

The same failure doesn't always look the same across platforms. Android and iOS pipelines differ in build tools, reporting formats, and failure patterns, so a rule tuned for one platform can silently miss an equivalent failure on another. We validate patterns against real production builds across platforms rather than assuming universal applicability — this also helps identify where platform-specific rules or evidence extraction are needed.

Improving evidence doesn't always simplify diagnosis, either — sometimes it reveals a deeper issue. A failure that initially looks like a missing dependency can, once more context is exposed, turn out to be a downstream configuration or environment problem instead. The broader lesson: better RCA isn't about collecting more logs, it's about collecting the right evidence and interpreting it in the correct execution context.

Observed Outcome

The RCA Agent is now live in production and has been validated against real failures rather than synthetic test cases, including dependency resolution errors, compilation failures, coverage regressions, and build and toolchain failures.

We also validated an AI-generated diagnosis by deliberately introducing a known bug. The agent identified the expected root cause down to the file and line number.

The production impact can be summarised across triage time and infrastructure recovery:

We are now collecting feedback from engineering teams through a lightweight feedback mechanism. So far, we have received around four pieces of direct stakeholder feedback and resolved 5–7 issues, including improvements to Issue Type tagging, classification, and alert content.

This feedback loop helps us continuously improve the agent based on real production usage and failure patterns, rather than relying only on synthetic test cases.

Conclusion

At roughly 800 builds and 190 failures a day, the RCA Agent now runs quietly in the background — analysing failures as they happen and giving engineers actionable context directly in Google Chat. Around 60% of failures are now resolved instantly, and average triage time has dropped from roughly 10–15 minutes to under a minute, saving an estimated 30–45 hours of combined engineering effort across the team each day.

One of our biggest learnings was that improving RCA is not only about improving the AI model. Some of the most important improvements were in the evidence pipeline—making sure the agent receives the right information and understands the execution context in which the failure occurred.

As a next step, we're exploring the addition of a memory layer to retain validated failure patterns and previous diagnoses. This can strengthen our existing Tier 2 approach by making previously analysed failure knowledge easier to reuse for recurring failures and reducing unnecessary AI analysis.

The goal was never to replace engineers who can diagnose a stack trace in seconds. It was to reduce the investigation effort for failures that aren't immediately obvious.

The model is only one part of the system. The real engineering is in getting the right evidence in front of it, knowing what it doesn't know, and verifying before trusting.

References:

• Agent Skills — Anthropic official docs
• Amazon Bedrock — What is Amazon Bedrock (official AWS docs)
• Jenkins Pipeline — official documentation

Join us

Scalability, reliability, and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels, and if solving hard problems with challenging requirements is your forte, please reach out to us with your resume at careers.india@halodoc.com.

About Halodoc

Halodoc is the number one all-around healthcare application in Indonesia. Our mission is to simplify and deliver quality healthcare across Indonesia, from Sabang to Merauke.

Since 2016, Halodoc has been improving health literacy in Indonesia by providing user-friendly healthcare communication, education, and information (KIE). In parallel, our ecosystem has expanded to offer a range of services that facilitate convenient access to healthcare, starting with Homecare by Halodoc as a preventive care feature that allows users to conduct health tests privately and securely from the comfort of their homes; My Insurance, which allows users to access the benefits of cashless outpatient services in a more seamless way; Chat with Doctor, which allows users to consult with over 20,000 licensed physicians via chat, video or voice call; and Health Store features that allow users to purchase medicines, supplements and various health products from our network of over 4,900 trusted partner pharmacies. To deliver holistic health solutions in a fully digital way, Halodoc offers Digital Clinic services including Haloskin, a trusted dermatology care platform guided by experienced dermatologists.

We are proud to be trusted by global and regional investors, including the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. With over USD 100 million raised to date, including our recent Series D, our team is committed to building the best personalized healthcare solutions — and we remain steadfast in our journey to simplify healthcare for all Indonesians.