From Events to Security Incidents: Leveraging Correlation in OpenSearch

Introduction

Modern security systems don’t fail because they miss signals — they fail because they generate too many. In large, cloud-first environments, security teams are flooded with alerts from identity systems, cloud platforms, and internal applications. While each alert may not be harmful in isolation, many lack the context needed to indicate whether a real security incident is unfolding, forcing security analysts to manually correlate events across tools and logs to understand what actually matters.

At Halodoc, our environment produces high volumes of security telemetry across identity systems, cloud infrastructure, and internal applications. While this data was available in OpenSearch, detections were largely built around single events and individual log sources.

This meant we could detect isolated signals — failed logins, blocked requests, or unusual API calls — but often lacked visibility into how these events connected across systems. As a result, suspicious behaviour spanning identity, cloud, and internal data access frequently required manual investigation and log correlation to determine whether signals were related to the same underlying activity.

This gap created investigation noise, slowed response times, and increased the risk of missing real attacks. In this post, we’ll share how we used event correlation in OpenSearch to evolve from event-level detection to incident-based security monitoring — linking signals across systems to surface real attacker behavior.

Why Investigation Noise Becomes a Security Risk

At scale, investigation noise is not just an operational inconvenience — it directly impacts security outcomes:

• Security analyst time is consumed by manual correlation of low-signal events, reducing focus on high-risk activity
• Mean Time to Detect (MTTD) increases as analysts spend time stitching together signals across logs and systems
• True attacks are easier to miss when meaningful activity remains fragmented across standalone events

What Is Correlation ?

Correlation in security detection refers to the ability to link multiple events across systems, log sources, or time windows to form a larger and more meaningful security narrative. Instead of treating alerts as independent incidents, correlation analyzes shared entities such as usernames, IP addresses, files, devices, and session identifiers to determine whether the events are related.

Common correlation patterns include associating failed authentication attempts with subsequent successful logins for the same account, or linking identity activity with downstream actions in cloud or data-access systems. In OpenSearch, correlation operates by matching these entities within a defined time window, transforming raw log events into structured security incidents that reflect attacker behavior more accurately than single alerts.

How Correlation Helps ?

Traditional SIEM detections operate using a single-event model: one event triggers one alert. While this works for simple threats, it often fails to surface multi-step attacks in real time. Modern adversaries do not achieve their objectives through a single action; they move laterally, escalate privileges, and access or exfiltrate data through sequences of events that span identity, network, and collaboration systems. When logs are analyzed in isolation, these sequences remain hidden, leading to visibility gaps, alert fatigue, and delayed detection.

For example, a blocked probing attempt from a WAF may seem noisy and benign, and a separate cloud login from the same IP may seem normal. However, when correlated, they indicate possible credential compromise something neither event reveals alone. Correlation solves this operational gap by connecting the dots between events and surfacing meaningful incidents automatically, reducing false positives and accelerating the SOC’s ability to detect attacks that would otherwise go unnoticed.

What Does Correlation Mean in OpenSearch?

In OpenSearch, correlation refers to the process of identifying related log events by evaluating shared attributes within a defined time window at query execution time. Rather than maintaining persistent state or attack graphs, correlation is implemented through search queries that group and evaluate events across one or more log indices.

We designed our correlation logic around three core dimensions. An incident is generated only when all three align:

• Entity
  The subject performing the action, such as a user ID, service account, IP address, or device.
• Behavior
  The type of activity observed, for example authentication attempts, file downloads, or configuration changes.
• Time
  A defined sliding window during which related events must occur, typically ranging from 5 to 30 minutes depending on the use case.

By requiring alignment across entity, behavior, and time, correlation filters out isolated noise and surfaces activity that reflects real attacker intent rather than standalone events.

Why OpenSearch Worked for Us

OpenSearch already sat at the center of our security observability stack. Extending it for security correlation allowed us to build incident-level detection without introducing additional platforms or complexity:

• Unified analytics across infrastructure, application, and security logs, enabling cross-domain correlation
• Flexible schemas for heterogeneous data sources, allowing identity, cloud, and app events to be analyzed together
• Query-level control over correlation logic, making detection behavior explicit and auditable
• Native integration with alerting and notification workflows, reducing operational friction

By leveraging OpenSearch’s existing capabilities, we avoided deploying a parallel SIEM system and significantly reduced operational overhead while improving detection quality.

Example Use Cases :

Now, let’s go through some of the use cases in detail.

1) Data Loss Prevention (DLP) Block + GDrive Download

Scenario 1 : User attempts to share a confidential file (blocked), then downloads same file via Drive to potentially exfiltrate externally.

Event A: A user is blocked from sending or sharing a sensitive file externally by a DLP policy.
Event B: Shortly afterward, the same user downloads or accesses that file through a different channel.
Match Fields: file.name + user.email
Time Window: 0–4 hours

Individually:

• Event A appears as a routine policy block.
• Event B appears as normal file access.

Together: Indicates a potential attempt to exfiltrate data through an alternate method after the initial attempt was blocked.

This sequence is frequently observed in insider threat and data leakage scenarios.

2) Suspicious Google Login + AWS Sensitive API Access

Scenario: User account shows suspicious activity in Google, followed by AWS access from the same IP.

• Event A (Google Auth): Suspicious login detected (new IP / geo / device).
• Event B (Security Hub / GuardDuty): Sensitive AWS API calls (GetObject, ListBuckets, DescribeSecrets) from the same IP.
• Match Fields: source.ip + time
• Time Window: 0–4 hours

Individually:

• Event A: Google flags login risk
• Event B: API activity appears valid

Together:
Indicates a possible compromise of the Google account, followed by use of the same access path to perform suspicious AWS activity.

How We Built Correlation in OpenSearch

Correlation detections were implemented using Query-Level Monitors in OpenSearch, enabling native, reliable multi-event analysis without external plugins.

Approach Used: Query-Level Monitor Correlation

• A single combined query is written to search across both log sources (for example, Google Auth logs and AWS Security Hub / GuardDuty logs).
• Events are grouped using composite aggregations on shared identifiers such as:
  • source.ip
  • user.email
  • userAgent
• Time-based filtering is applied within the query to ensure events fall within the defined correlation window.

Correlation Logic

• A script is used within the aggregation to:
  • Validate the presence of Event A (suspicious activity in Google)
  • Validate the presence of Event B (suspicious AWS API activity from the same IP)
  • Confirm both conditions are met within the same bucket

Alerting & Context

• Mustache templates are used in notifications to dynamically include:
  • User identity
  • Source IP
  • Event timestamps
  • Matched indicators from both log sources

Alerts are triggered only when both event conditions exist, reducing noise from standalone events.

Note : Correlation only works when all related logs have arrived in OpenSearch. Different systems send logs at different speeds, some arrive in minutes, while others can take one to two hours. If Event A appears quickly but Event B is delayed, the monitor cannot connect them yet. The alert is therefore generated only after both logs are available and the scheduled query runs. For this reason, correlation is near-real-time rather than instant, and the delay usually comes from log ingestion time rather than the detection logic itself.

Impact: MTTD (Mean Time To Detect) Reduction

Prior to introducing correlation-based detections, identifying multi-step security incidents often required manual investigation across multiple tools and log sources. In these cases, the time from the first relevant security signal to an actionable detection commonly exceeded several hours.

After deploying rule-based correlation using OpenSearch query-level monitors, we observed a measurable reduction in detection time for a subset of multi-signal attack patterns. In early measurements, the average time from the first relevant event to a correlated alert was reduced from approximately five hours to around ninety minutes , taking log-ingestion delays into account.

This represents an estimated reduction of roughly seventy percent in MTTD for the evaluated scenarios. These results reflect initial observations and are dependent on factors such as log ingestion latency, correlation window configuration, and monitor execution frequency.

This ultimately led to:

• Reduced investigation noise, as multiple related log events — and in some cases duplicate alerts — are consolidated into a single correlated finding
• Faster triage, since analysts receive unified context and timelines instead of manually stitching together isolated signals
• Lower false positives, increasing confidence in detection outcomes and prioritization
• Improved visibility into previously hidden activity, where low-signal events that appeared benign in isolation are now surfaced as meaningful incidents when viewed together

Post implementation, thousands of raw log events generated across identity, cloud, and application systems are now consolidated into a much smaller set of meaningful correlated events, representing actual security-relevant activity rather than isolated signals.

While correlation does not eliminate the need for security analyst investigation, it enables suspicious activity spanning multiple systems to be surfaced earlier and with greater context, improving visibility into complex incidents and accelerating response workflows.

Conclusion

Correlation transforms detection from isolated alerts into contextual security incidents that reflect how real breaches unfold whether driven by insider risk, compromised internal accounts, or external attackers leveraging internal access. By linking events across logs and time, OpenSearch enables SOC teams to uncover hidden attack paths, reduce blind spots, and significantly accelerate detection.

This capability is essential for modern cloud-first environments where identity, internal data access, and network activity span multiple systems, but the security story often unfolds across all of them.

References

https://opensearch.org/platform/event-correlation/
https://opensearch.org/blog/correlating-security-events/
https://opensearch.org/docs/latest/observability/alerting/

Join us

Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.

Bug Bounty

Got what it takes to hack? Feel free to report a vulnerability in our assets and get yourself a reward through our bug bounty program. Find more details about policy and guidelines at https://www.halodoc.com/security 

About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. We recently closed our Series D round and In total have raised around USD$100+ million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.