Beyond Executor Slots: Building a Memory-Budget Allocator for Jenkins CI

At Halodoc, delivering reliable digital healthcare experiences depends not only on the applications we build, but also on the engineering systems that help us build, test, and release them safely.

As our mobile platforms and CI workloads grew, we encountered a limitation in how Jenkins scheduled builds across shared Mac workers. Jenkins could identify an available executor, but it could not determine whether the underlying worker had enough allocatable memory to complete a resource-intensive build reliably.

This gap allowed multiple memory-heavy workloads to run on the same worker, causing resource contention, out-of-memory terminations, and wasted build time. The challenge was not simply a shortage of executors, it was the absence of workload-aware admission control.

To address this, Halodoc’s engineering team built a memory-budget allocation layer for Jenkins. The solution assigns an expected memory budget to each workload, atomically reserves capacity on an eligible worker, and safely releases that reservation after execution.

In this article, we explain the architecture behind the allocator, how we handled concurrent reservations and failure recovery, the trade-offs we encountered, and the operational impact observed after introducing it.

A Free Executor Did Not Mean a Healthy Worker

Jenkins models capacity as executor slots, a machine either has a free slot or it doesn't. That abstraction works fine until the workload stops being uniform. Mobile CI breaks it.

iOS and Android builds don't behave like a typical job queue. iOS requires macOS – Apple's toolchain won't run anywhere else and our Android builds share the same on-premise Mac fleet for signing and consistency. These are physical machines in our own rack, not elastic cloud instances you can scale on demand. Capacity is fixed.

What Jenkins understood: whether an executor slot was free.
What Jenkins did not understand: how much memory that machine actually had left.

The gap between those two things is where the failures lived. A machine with 2 GB remaining would accept a new build without hesitation as the slot was open. A full Android Gradle build needs 8+ GB. An iOS pipeline runs three parallel stages simultaneously quality gate, staging, production each requiring independent memory headroom. Jenkins had no model for any of this. The result was predictable: builds started, ran for 15–20 minutes, then died. OOM kills with cryptic errors. Developers retrying on the same saturated machine. No visibility into why.

Adding more executors would not have helped. More slots on an already memory-pressured machine means more concurrent builds competing for the same shrinking pool. The problem was not throughput, it was overcommitment. Jenkins was promising capacity it could not honour.

We needed the scheduler to understand memory, not just executor slots.

Why the Executor Model Broke Down

Not all mobile builds are equal – and that variance is exactly what made the executor model break down.

Jenkins assigned all three the same way: one executor slot, one machine, no memory context. A lightweight library job and a full regression suite looked identical to the scheduler. When they landed on the same machine simultaneously, the regression suite would consume whatever memory the library job left behind - sometimes enough, often not. The OOM kill would come 15 minutes in, with no indication that memory pressure was the cause.

A Shared Ledger for Memory-Budget Reservations

The core idea: maintain a shared record of each worker's allocatable budget and active reservations, and make every allocation decision against that record atomically.

Three components make this work:

Workload budget configuration - a JSON file that maps each service to an expected memory tier. Before a build starts, the allocator looks up how much capacity this workload needs. This is a declared budget, not a runtime measurement. Worker-capacity ledger - a DynamoDB table with one record per Mac agent, tracking allocatable capacity and current reservations:

Allocator reads the ledger, selects a worker with sufficient remaining budget and an open reservation slot, and performs a guarded conditional write. If two builds race for the same worker simultaneously, only one succeeds and the other moves to the next candidate. When the build finishes, pass or fail, the reservation is released and the budget returned.

That's the whole idea. Everything else is making it robust.

Calibrating Workload Budgets

Tier boundaries were not guessed. Before assigning any values, we observed six months of OOM incident history across all Mac agents and build failures where the kernel terminated the process due to memory pressure. From that we identified which workload categories were consistently involved and at what approximate consumption levels.

What we measured: memory-pressure events and OOM kills correlated against build type and service category.

How we set the boundaries: for each workload category, we identified the consumption level at which OOM kills stopped occurring and added a safety margin.

The Allocation Lifecycle

The allocator runs outside the Mac executor pool and builds wait for a reservation without consuming an executor slot on any Mac.

1. Resolve the workload budget - look up the service in the budget config to determine the memory tier required.
2. Read candidate worker capacity - scan the ledger for current available budget and active reservation counts across all workers.
3. Exclude ineligible or unhealthy workers - drop any worker below the required budget, at its reservation limit, or currently quarantined from a failed health check. Sort remaining candidates by least available budget first — tight packing.
4. Attempt an atomic reservation - write a conditional update against the ledger. If the worker's state changed since the scan, the write is rejected and the next candidate is tried. A rejected write is not an error, it is the race-condition handler.
5. Run the workload - once reserved, the build executes on the selected worker.
6. Release the reservation idempotently - on completion, the budget is returned and the reservation count decremented. A ceiling guard prevents the ledger from exceeding physical capacity regardless of how many times release is called.

If no eligible worker exists at step 3, the allocator waits and retries. The build queues, it does not fail.

Preventing Double Booking

The tight-packing sort narrows the candidate list, but it doesn't prevent a race. Two builds can read the same ledger state simultaneously, both decide mac-2 has enough budget, and both attempt to reserve it. Without coordination, both succeed and the machine is overcommitted.

If two builds race for the same worker, DynamoDB applies one write and rejects the other with a ConditionalCheckFailedException. The rejected build moves to the next candidate and retries. The conditional update prevents two successful reservations from consuming the same available budget, without requiring a separate distributed lock.

The failure case is handled explicitly, not assumed away the allocator treats a failed conditional write as a signal to move on, not as an error.

Failure Recovery

Every reservation must eventually be released. The system handles four exit paths

A ceiling guard is applied on every release, remaining budget cannot exceed physical capacity, preventing double-releases from inflating the ledger.

One known gap: if an agent is catastrophically lost before either release path fires, the reservation stays occupied until the next reconciliation cycle. Automated lease expiry to handle this case is on the roadmap.

Heath Checks, Quarantine and Observability

Before finalising a reservation, the allocator verifies access to critical CI dependencies and source control, artefact repositories, and code analysis services from the candidate worker. A worker that cannot reach them would produce a failed build regardless of memory availability, so it is excluded before the reservation is committed.

A worker that fails the connectivity check is temporarily quarantined for the duration of the current allocation attempt. It is not permanently removed the next build cycle re-evaluates it from scratch. Recovery is automatic: a worker re-enters the candidate pool as soon as it passes a connectivity check. No manual intervention is required unless the failure is persistent, in which case SRE is alerted.

Every allocation event is written to an audit log — worker, workload category, tier, budget reserved, before and after ledger state, and stage name. This gives a full trace of how capacity moved through the fleet and is the primary input for quarterly tier reviews.

Operational Impact and What We Can Attribute

We compared two 30-day windows — before and after the allocator went live. Before drawing conclusions, two confounds worth naming: build volume grew 47.9% in the same period, and mac-4 was added mid-window, increasing raw fleet capacity. Neither variable is controlled for. The comparison shows the system operating under a heavier, growing load — not a fixed workload measured twice.

Pass rate and failure rate moved in the right direction. We are not presenting these as allocator outcomes and the build mix changed, a new machine was added, and other concurrent changes cannot be ruled out. These are directional observations.

On duration: average build time increased 4.1 minutes (+19%). Builds now wait for a worker with sufficient budget rather than starting immediately and failing mid-run. The wait is upfront and bounded. The previous failure was mid-build and silent. That tradeoff is intentional, though we have not yet instrumented pre-build queue wait as a separate metric to prove it directly.

Trade-offs and Current Limitations

Budgets, not measurements. The ledger tracks reserved memory, not actual OS consumption. A misconfigured or leaking build can still OOM and the system prevents contention between correctly classified builds, not individual failures. Classification accuracy matters. An incorrect or stale tier assignment leads to underutilisation.

Builds wait. Trigger-to-result time increases under load. HIGH-tier builds have no priority and they wait in the same queue as LOW-tier ones. Maximum wait thresholds need an explicit policy. The ledger is a CI control-plane dependency. Allocation fails closed when ledger state cannot be verified. DynamoDB availability is now a CI availability concern.

Designed for a small fleet. Works well at 4 agents. At larger scale, allocation latency and tier staleness need revisiting.

Conclusion

Jenkins executors remain useful, but they are incomplete when workloads have substantially different resource demands. Introducing a shared reservation ledger allowed us to make memory budgets explicit and coordinate allocations atomically across pipelines.

The transferable lesson is not that every team needs DynamoDB. It is that constrained resources should be represented explicitly, reservations should be atomic and recoverable, and success should be measured against the failure mode the system is designed to prevent.

Our implementation still has boundaries: it schedules configured memory budgets rather than real-time system pressure, and the comparison period included both workload growth and additional hardware. Within those boundaries, the operational data indicates that the approach improved CI stability under increasing demand.

References

Join us

Scalability, reliability, and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.

About Halodoc

Halodoc is the number one all-around healthcare application in Indonesia. Our mission is to simplify and deliver quality healthcare across Indonesia, from Sabang to Merauke.
Since 2016, Halodoc has been improving health literacy in Indonesia by providing user-friendly healthcare communication, education, and information (KIE). In parallel, our ecosystem has expanded to offer a range of services that facilitate convenient access to healthcare, starting with Homecare by Halodoc as a preventive care feature that allows users to conduct health tests privately and securely from the comfort of their homes; My Insurance, which allows users to access the benefits of cashless outpatient services in a more seamless way; Chat with Doctor, which allows users to consult with over 20,000 licensed physicians via chat, video or voice call; and Health Store features that allow users to purchase medicines, supplements and various health products from our network of over 4,900 trusted partner pharmacies. To deliver holistic health solutions in a fully digital way, Halodoc offers Digital Clinic services including Haloskin, a trusted dermatology care platform guided by experienced dermatologists.
We are proud to be trusted by global and regional investors, including the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. With over USD 100 million raised to date, including our recent Series D, our team is committed to building the best personalized healthcare solutions — and we remain steadfast in our journey to simplify healthcare for all Indonesians.