Sentinel MCP: A Deterministic Agentic Testing Workflow
At Halodoc, a fix for a production issue can now be drafted within minutes, and a whole feature can be built from its product requirements document (PRD) within hours. Most of that code is co-written with AI, along with the tests that validate it. But there's a caveat:
Problem 1: Tests written by an AI agent that reads the code tend to verify exactly what the code does, not what the requirement asks for.
Problem 2: When an agent says it has fully tested a change, engineers can't deterministically tell whether it actually ran every test it reported, skipped a step, or hallucinated the result.
How do we make agentic testing deterministic, so that every run goes through the same required checks and is judged by the same rules?
Our answer is Sentinel, a deterministic agentic testing workflow, shipped as a Model Context Protocol (MCP) server inside a plugin our engineers already use. When given a production incident fix, a merge request or a PRD, Sentinel runs the targeted testing that the change needs. AI agents do the reading, planning, and reproducing; deterministic gates validate every step and compute the verdict, with the evidence attached. What the agents write can vary from run to run or model to model; the steps and the rules that judge it don't.
Goal: An autonomous quality verdict we can rely on for every backend merge request and every backend PRD at Halodoc.
Before breaking the workflow down step by step, it is worth looking at how testing at Halodoc evolved.
The evolution of testing
Before Sentinel, testing a fix at Halodoc went through four phases, and each one fell short in a different way.
Sentinel began as testing agents chained in a fixed order, and its early runs showed that wasn't enough: it marked one fix Ready for Prod without ever reproducing the original failure, and called a deploy healthy 29 seconds into an 8.5 min rollout. So Sentinel became an agentic workflow: the path is fixed in code, agents work inside each step, and code validates each step's output before the next one runs.
Rules for deterministic agentic testing
Code can only validate a step if it knows what counts as proof, so we wrote down what it would take to trust a test verdict we hadn't produced ourselves. The list grew each time a real run went wrong, and every rule on it is now enforced in code.
- The requirement is the judge, never the code.
- Models gather evidence; rules decide. The verdict is computed by plain code from the evidence each step recorded. No model makes the final call.
- A judgement can hold a change back but never let it through. A model's opinion is enough to stop a change. Passing needs something observed: a reproduced failure, a replayed request or a measurement.
- Worst news first. A live error outranks any number of passing tests, and a deploy nobody saw land can't pass.
- "Unknown" is a result. A check that couldn't run is reported as skipped, never ok, so it can't be the reason a change passes. Unknowns go to a person.
- Partial evidence can prove a change unsafe, never safe. A break in a partial list of callers is still a break; a "nothing breaks" verdict has to hold for every caller.
- A test has to be able to fail. Test coverage only tells you a line ran during a test, not that the test would notice if that line were wrong. So Sentinel breaks the changed code on purpose in isolation to test the quality of the test: a test that still passes is flagged as weak, never accepted.
- Every requirement maps to a test, or it's reported as missing.
- Prove it with a real reproduction. Sentinel recreates the failed production request on stage, from how it failed, and replays that exact request before and after the fix. It runs against the service's real build and queries the database on its own, instead of trusting a mock or a test written after the fix is merged.
- Sentinel's agents are scored on their testing, not on the merge request (MR) getting merged. A run that catches a bad fix is a successful run for Sentinel, so it gains nothing by letting an MR through.
- Sentinel merges to stage, nowhere else. Production is read-only to it, and the merge to master still stays with an engineer.
- If it isn't on disk, it didn't happen. Every step writes its result to the run's record. That record is what a verdict cites as source, and what a stopped run resumes from. So a verdict can't cite a result that was never recorded, and a resumed run doesn't redo finished steps.
How Sentinel is put together
- Under the hood, Sentinel enforces those rules through a library of 66 MCP tools, each doing one job and testable on its own.
- 29 agentic tools call a model, so their output can vary between runs, but the steps they run in and the rules that judge them don't.
- 37 plain-code tools and gates return the same answer every time.
- Each mode chains these tools into a fixed flow.
- Each agentic tool runs its own agent loop: it calls tools, reads what comes back and repeats until it can answer in the fixed schema. A flow can call the same tool more than once, such as reproducing a failure before and after the fix, and plain code checks every answer before the next step runs.
- Every tool is also exposed on its own, so an engineer's coding agent can call just one, such as review_fix for a code review or reproduce_5xx to replay a failure on stage, without running a whole flow.
- Engineers trigger Sentinel through the MCP server in their coding agent, from a command line, or through a dedicated Sentinel CI pipeline that runs unattended. Our auto-fix pipeline, which drafts fixes for production errors twice a day, triggers it after each of its own runs.
A few of the critical components:
- Code intelligence: Works out who can reach a changed line, from a call graph parsed with tree-sitter (Java, Python and Go) and dependency graphs of which services reach it over HTTP, Kafka or gateway routes. No model is asked who the callers are.
- Data and requests: Databases go through HaloDB, our in-house wrapper around the open-source DBHub MCP server: one gateway to every Halodoc database, with credentials provisioned automatically.
- Decision layer: Plain rules, plus a model's second reading for runs the rules can't settle. That reading can hold a change for QA or a developer, but never pass, merge or revert one.
- Adapters: Connectors within the MCP to where the evidence lives: our observability tool for logs and traces, EKS pods, Jenkins builds, GitLab and OpenProject.
Decisions and trade-offs
A test index instead of search. Asking a model to find the test, or searching every automation repository, or scanning our internal Sourcegraph for the tests that cover a change was slow and gave a different answer each time. So we built a periodically updated index: 27,773 backend tests across 91 repositories, mapped to the 4,766 endpoints they exercise. Finding coverage is now a fast lookup, and a text match alone never counts as coverage.
Orchestrate the skills we already had. Our engineers already had AI skills that plan tests and write them across our four layers (unit L1, component L2, integration L3, end-to-end L4). Sentinel calls them as MCP tools and owns only the order, the checks and the verdict, so it improves automatically whenever they do.
Baseline first, then replay. Before the fix lands, Sentinel captures the failing request and response, how many times each database statement runs, and the chosen test results. After the fix, it replays the same thing and compares.
Targeted testing. Sentinel doesn't always run a feature's whole bloated test suite. It picks the tests that cover the change and generates its own TestNG suite XML on the fly, including every test those depend on.
A fixed schema for every model answer. Each agentic step must answer in a fixed JSON schema. An answer that doesn't validate is a failed step, so a half-empty answer can't pass as a confident "no".
Telemetry on every step. Each step reports its duration, model calls and outcome to OpenTelemetry, tagged by endpoint and by who started the run.
Incident modes: testing a fix from its failures
Sentinel's five incident modes share those rules, tools and decisions. What differs is the question each one asks of a change, and what counts as an answer.
Here is the 5xx mode end to end, with the Sentinel MCP tool behind each step. Each row is one step: tools that call a model sit on the left, and plain-code tools and rules on the right, so you can see where a model hands over to code and back. The other modes chain their own tools the same way, and their steps are in the toggles below.
5xx: proving the fix made the error go away
When a production 5xx comes in, our auto-fix pipeline drafts a fix and Sentinel tests it in the steps shown above. The rules then decide where the fix goes:
- Ready for prod: the failure reproduced before the fix, and the replay passes after it.
- Back to the developer: the failure is still there after the fix.
- A stage revert, for a person to merge: a request that worked before the fix fails after it.
- To a person: the failure couldn't be reproduced.
- To QA: the replay couldn't run, so only the targeted regression tests ran, as a sanity check.
Of the auto-fix pipeline's production 5xx fixes that reached Sentinel, 39 of 40 got a verdict.
Case study: In one run, Sentinel traced a tracker API error to data owned by another service, reproduced the exact error on stage, and showed that the drafted fix would never have resolved the error.
4xx: differentiating a correct 4xx rejection from a real bug
Most 4xx errors are the system doing its job. Before we built 4xx mode, we went through 216 closed 4xx tickets: about 73% were expected errors and only about 2% needed a server fix. So Sentinel has to work out what the resolution should be before anything ships:
- Reproduce: it replays the request on stage, read-only. Only the exact status counts, and an inconclusive result goes to a person.
- Explain: it sorts the error by why it exists, and each reason has its own next step: an error by design is closed as expected behaviour, one relayed from another service goes to the service where it started, one avoidable by the caller comes with what the caller should change, and one from a partner's own rule is coordinated with the partner. Only a genuine server defect can lead to a fix.
- Fix: a server defect goes on to stage only with a signed-off review and safe callers, and the same request is replayed for the verdict.
Case study: On a listing page API, Sentinel tried page sizes on stage: 10 and 100 returned products, while 101, 150 and 1,000 got the same validation message every time. It closed the ticket as expected behaviour and suggested capping the page size at 100 in the frontend app.
Kafka: how to test a consumer fix with one event and two builds
Kafka consumer fixes are hard to test by hand and can conflict if tested on stage, so Sentinel builds its own environment to test the fix:
- Set up a private broker: a local broker and schema registry with every topic the service uses.
- Build the service twice: with and without the fix, and check that the builds differ only by the change.
- Replay one event: boot each jar against stage with only Kafka redirected, and produce one event the way a real producer would.
- Decide: without the fix, the event has to fail inside the changed code; with it, the failure has to be gone.
- Clean up: the harness is torn down and the seeded data reverted. Nothing is published to stage Kafka, so the fix is tested before it ever reaches stage.
Case study: In one consumer flow, a failed save had been swallowed, and an event was still sent for a record that was never saved, so the next service's lookup returned a 404. Sentinel reproduced the 404 without the fix and showed it was gone with the fix. It also caught a second event still being published for the same failed row.
Slow query: proving a faster query returns the same rows
A rewritten query is only a fix if it returns exactly the same rows. Sentinel checks that in these steps:
- Connect: a read-only connection to one production schema, with every statement capped at 9.5 seconds so as not to cause a slow query during testing.
- Pair the queries: the model writes matched pairs of old and new queries.
- Compare the rows: the database itself compares each pair, reducing each result to three numbers: a row count, a sum of row hashes, and an XOR of row hashes. Duplicate rows cancel out in an XOR, and the sum catches that. Rows leave the database only when the numbers differ, and then only as a small sample.
- Compare the plans: the index chosen, any full scan or sort, and the optimiser's own cost estimate, to confirm the rewrite really is cheaper.
- Decide: a rewrite passes only if all three numbers match for every pair, with at least one pair on real production values, and a mismatch counts only if it repeats. If the original plan was already clean, Sentinel reports that the query was never the problem.
Case study: On its first live run, Sentinel was given two rewrites of a real slow query. The correct one returned 7 rows on both sides; the other widened a filter and returned 145,574 rows against the original's 7, so Sentinel marked it non-equivalent.
API latency: why count queries instead of timing them
API latency mode has no latency target. Instead, it works through these steps:
- Diagnose: production traces show why the endpoint is slow. If the change doesn't touch that, Sentinel stops there.
- Build a real request: from the endpoint's contract. Some causes only show up at scale, such as a full table scan or an N+1 query, so for those, Sentinel first fills the stage database with synthetic rows shaped like production, never copied from it, and deletes them afterwards.
- Measure before and after: it replays the same request on stage before and after the fix, checks that the response hasn't changed, and counts how many times each database statement runs.
- Decide: the statement count decides whether the fix helped, because it repeats from one machine to the next and timing doesn't. Timing is a check on top: it's ignored inside the endpoint's normal noise (15% or its usual run-to-run spread, whichever is larger), but a slowdown beyond that outranks a better count and fails the fix.
- Hand over: if the fix rewrites the one query where most of the time goes, the run hands over to slow-query mode.
Case study: A search API fix passed review and 2,612 CI tests. On stage, its main statement ran nine times instead of five, and the endpoint was 62% slower instead of being faster. Sentinel marked it as not improved and sent it back to the developer, so it never reached production.
Here is that API latency case study as Sentinel ran it, from the real run record:
The output is always a verdict with receipts
Whichever question a mode asks, the answer arrives the same way: a verdict on the ticket and in the team's chat, linked to the full run report. A 5xx verdict that clears every check looks like this:
Each number is a receipt you can open in the run report:
1. Correct: the sign-off from the review_fix tool, its confidence, and the source lines it read.
2. Safe for all its callers: the caller list from the resolve_impact tool, with a ruling on each caller from the review_caller_impact tool.
3. Reproduced on stage: the exact request reproduce_5xx sent, the 500 that came back, and whether it matches the fix's root cause.
4. Gone after the stage deploy: a healthy stage deploy, then the same request replayed, and its new status.
5. No existing test changed result: the before-and-after diff of the chosen tests from run_targeted_tests.
The verdict itself is computed by decide_disposition, in plain code, with no model call. Merging to master stays with the engineer, who now reads a verdict and its evidence instead of testing the fix again.
PRD mode: testing a feature from its requirement
Incident modes ask one question about one fix. PRD mode asks a bigger one: does a whole feature do what its requirement says? Give it a PRD, a technical change document (TCD), a feature merge request, or a few lines of intent, and Sentinel runs the testing lifecycle across every service the feature touches. It's the one mode that isn't autonomous: a person stays in the loop wherever judgement is needed.
A feature moves through eight stages:
- Pin the requirement: the exact version of the PRD and TCD, so every judgement is made against a fixed source.
- Agree the intent: quoted word for word from a confirmed document, or proposed by Sentinel and agreed by a person.
- Ask once: a question already asked, answered or decided is never asked again.
- Plan every layer: cases for each requirement from L1 to L4, checked against the test index. Where the PRD, TCD, code and merge requests disagree, a person decides which side stands.
- Write the tests: service by service, through our layered testing skill.
- Prove each test can fail: mutation testing on the lines the feature changed.
- Run everything the way CI does: each affected service's full suites, plus a check of the stage logs.
- Compute the verdict: Signed off, Conditional, Blocked or Not evaluated, worked out from the record with no model call.
Every decision lands in the feature's record, so a run can pause for days and anyone can resume it with "retest this feature". An approval counts only if it matches the person's own words, not the agent's summary of them.
PRD Case study: the VIP priority notification
One of the first PRDs we ran through Sentinel was a VIP priority notification, which alerts clients when a VIP customer's records go through certain status updates. Sentinel read 62 requirements, raised 35 questions and wrote 125 tests, with every requirement traced to its tests.
The services' full suites, nearly 17,500 tests, all passed. Sentinel still found what they had missed:
- Full coverage, no checks. Existing tests ran every line of a data transform but never checked its output. Sentinel planted a small bug in the transform: its new tests caught it, and the old ones had nothing to fail.
- Tests that never checked the notification. Three existing tests still passed with the notification removed; Sentinel's new tests failed without it.
- An untested safety check. The Kafka event must publish only after the database save, or other services act on a change that never happened. After the stage merge, no existing test reached that check, so Sentinel made it the top-priority test case.
- Tracking that didn't work. None of the 255 notifications sent on stage carried the delivery ID the design needs.
- Six bugs: two fixed before release, two older ones sent to their owning teams, and two left to a product decision.
Sentinel blocked the release because the feature had started publishing four new statuses to Kafka topics other services already consume, and no test showed those consumers would ignore them. The team made the fixes and reran Sentinel, which picked up where it had stopped and signed off on the feature.
Outcomes
Here is where Sentinel stood in September 2026, when we finished building it:
What's still hard
- A one-off production failure, such as a pod restart or a brief timeout, can't be reproduced on stage, so Sentinel leaves that call to a person.
- Recreating the failing production state on stage is the hardest step: it can mean setting two columns on one row or building a chain of records across several service databases.
- Sentinel tests backend services only, so errors or features that exist in our web and mobile apps aren't covered yet.
- Sentinel blocks a fix on the strongest evidence it can gather, and none of the 20 fixes it blocked has been merged unchanged so far, but proving a block was wrong independently is still beyond what one run can show.
Conclusion: How do you make agentic testing deterministic?
What makes Sentinel's verdicts dependable is the design around the testing agents: a fixed order of tool calls, a guardrail verdict after every step, and rules in plain code scripts that decide what the AI-provided testing evidence proves. None of that depends on which model does the reading and writing, so we can change the model behind any step without moving the definition of a pass. If you're building something similar, try techniques that reproduce the failure before and after the fix, capture automation test baselines just before the fix lands, break the code on purpose to check that every new test can fail, and use the other deterministic checks above to verify each error.
When AI writes both the code and the tests, a passing build only tells you that your tests pass. It can't tell you whether they are the right tests, or whether every check really ran. That needs testing that is deterministic and judged by the requirement, which is what we built Sentinel to do.
References
- Model Context Protocol
- tree-sitter Parser Generator
- DBHub - database MCP server
- Agentic AI Quality Engineering
- Mutation testing
Join us
Scalability, reliability, and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels, and if solving hard problems with challenging requirements is your forte, please reach out to us with your resume at careers.india@halodoc.com.
About Halodoc
Halodoc is the number one all-around healthcare application in Indonesia. Our mission is to simplify and deliver quality healthcare across Indonesia, from Sabang to Merauke.
Since 2016, Halodoc has been improving health literacy in Indonesia by providing user-friendly healthcare communication, education, and information (KIE). In parallel, our ecosystem has expanded to offer a range of services that facilitate convenient access to healthcare, starting with Homecare by Halodoc as a preventive care feature that allows users to conduct health tests privately and securely from the comfort of their homes; My Insurance, which will enable users to access the benefits of cashless outpatient services more seamlessly; Chat with Doctor, which allows users to consult with over 20,000 licensed physicians via chat, video or voice call; and Health Store features that allow users to purchase medicines, supplements and various health products from our network of over 4,900 trusted partner pharmacies. To deliver holistic health solutions in a fully digital way, Halodoc offers Digital Clinic services, including Haloskin, a trusted dermatology care platform guided by experienced dermatologists.
We are proud to be trusted by global and regional investors, including the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. With over USD 100 million raised to date, including our recent Series D, our team is committed to building the best personalized healthcare solutions, and we remain steadfast in our journey to simplify healthcare for all Indonesians.
