Automating Event Validation for Analytics: A Practical Guide for CleverTap

Manually verifying analytics events generated by tools like CleverTap can be time-consuming and prone to errors. At Halodoc, we tackled this challenge by automating the entire process, making event validation faster, more accurate, and efficient.

This blog will introduce the common challenges in analytics event validation, outline how automation provides a scalable solution, and demonstrate a practical implementation using CleverTap as a case study.

Why Analytics Event Validation is Critical

Testing Analytics events is essential not only during the initial integration but also post-integration to catch potential regression issues. Key reasons for its necessity include:

• Misconfigured Events: Changes to the application may disrupt event tracking.
• Data Loss: Key data fields like user IDs can fail to pass correctly.
• Incorrect Event Triggers: Events may fail to fire or trigger at inappropriate times.
• Duplicate Events: New features can lead to duplicate logging of events, skewing analytics.
• Profile Mismatches: Events can mistakenly link to incorrect user profiles.
• Server-Side Issues: Analytics tools may experience server-side errors, including delayed event processing, partial data responses, or API outages. These issues can impact event validation accuracy and should be promptly identified and escalated to the tool's support team for resolution.
• Regression Issues: Updates can unintentionally break previously working events.

Regular validation, especially after updates, minimizes these risks and ensures data integrity.

Automation as the Solution

Initially, verifying analytics events required manually checking each event on the dashboard, navigating through user profiles, and comparing event logs against expected outcomes. As event tracking scaled, this approach became inefficient, time-consuming, and prone to human error.

Automation resolves these inefficiencies by:

• Ensuring event accuracy and consistency.
• Reducing manual effort and error-prone processes.
• Scaling event validation for complex user journeys and multiple platforms.

At Halodoc, we automated analytics validation for CleverTap, our chosen analytics tool, by leveraging its APIs. This streamlined event testing across web and app platforms and can be adapted to other analytics tools with similar API capabilities.

Prerequisites for Automation

Before implementing automation for analytics event validation, ensure the following:

1. Functional Automation Setup: Have a working test automation framework for web or mobile platforms, such as Selenium, Playwright, or Appium.
2. Analytics Tool Integration: Ensure your analytics tool (e.g., CleverTap) is integrated into your application and events are manually verified. For CleverTap integration you can get started from here.
3. Staging Environment: Set up a controlled staging environment that replicates production conditions for the application and setup a test project on your analytics tool for tracking the events on this environment.

Event Data Retrieval Approaches

There are multiple ways to retrieve event data depending on the analytic tool:

• Data Export (via S3 or Data Warehouse): Some analytics tools provide the ability to bulk export event data to cloud storage services (e.g., AWS S3). This can be useful for analysis in business intelligence (BI) tools or for long-term storage. However, this method is typically slower than real-time API queries.
• Analytics Dashboard UI Automation: Although less efficient, automating the dashboard UI is another option. By navigating through the dashboard and filtering events based on user profiles, events can be validated. However, this method is slower compared to API-based solutions, and event data may not be instantly reflected on the dashboard.
• Analytics APIs: Most analytics platforms offer APIs that allow programmatic access to event data. These APIs are the most efficient method for validating events in real-time, offering flexibility and speed in retrieving and validating event properties.

The API approach that we implemented for CleverTap, is designed to be adaptable to any analytics tool with API access. It ensures that event validation is fast, scalable, and platform-agnostic.

Getting Started with the CleverTap Event APIs

1. Obtaining API Credentials

• Log into your CleverTap Account
• Switch to your Test Project
• Click Settings on the left-side dashboard.
• Click on Project option
• Save the values of Project ID and Passcode

These credentials are required to authenticate API calls to CleverTap’s services.

2. Identify CleverTap Account Region

CleverTap's APIs are region-specific, so you need to select the appropriate base URL for your account’s region. This is critical to ensure the API calls are routed correctly.

You can either goto Settings>Project>Region to see your region or you can determine it from the dashboard url as follows:

For the examples below, Singapore is the chosen region.

3. Generate Cursor with POST API

Send a POST API request to obtain a cursor for a specified event within a one-day window. This cursor acts as a unique identifier, enabling access to user profile event data.

Parameters
batch_size: Number of event entries to retrieve in 1 API call
X-CleverTap-Account-Id: CleverTap account ID obtained from dashboard
X-CleverTap-Passcode: CleverTap passcode obtained from dashboard

Body
event_name: Name of the event being tested
from: Time window to query event trigger
to: Time window ending limit

This request returns a cursor specific to the event and time window, covering all user profiles.

Challenge: Duplicate user profiles maybe present in the response, especially when parallel tests are run or multiple users share the same account. This can cause confusion when trying to identify the most recent event.
Solution: To handle this, use the last_seen timestamp in the event data to identify and validate the most recent profile-event pair to occur within the specified time window.

4. Retrieve Event Data with GET API

Using the cursor generated in the earlier step, send a GET API request to obtain event details and attributes. The response provides information on the events triggered, based on the cursor.

• Ensure to handle any pagination with the next_cursor if more events need to be retrieved i.e if the event is not present in the first cursor

Challenge: CleverTap’s API responses return events and properties in separate objects, making it harder to correlate them together.
Solution: Link the events and their properties by associating them with the common user profile node.

5. Result Validation

These APIs need to be hit at the end of a desired functional flow leading to a page visit or user action, with the corresponding event name acting as the assert. If any events are missing from the dashboard i.e not triggered, the API response will also not have the event matching to that profile within that time window and hence the test step will fail for that event, allowing the tester to raise an issue and contact application developer/tool support if discrepancies arise.

Algorithm for Automated Event Verification

1. Record Execution Start Time

Start by recording the timestamp at the beginning of your test execution. This timestamp along with the test ending timestamp forms the window which will serve as a reference to verify that events are triggered within the correct window.

2. Suspend Test User to Handle Profile Merging

Challenge: CleverTap’s profile merging feature merges events from pre- and post-login profile states. These merged events will not appear in the API response but will be visible in the CleverTap dashboard. This can cause missing or incorrect event data when multiple test runs use the same test user.

Solution: To avoid profile merging discrepancies, suspend the test user between test executions and reactivate them just before the test. This ensures a fresh profile for each test run and prevents pre- and post-login profiles from being merged, maintaining event data integrity.

3. Execute the functional test steps using any automation tool (Selenium, Playwright, Appium). You can abstract this using any BDD tool as well like Cucumber.

4. Call the CleverTap event verification step that triggers the API calls, passing the phone number (or any other user profile identifier) and a data table with eventType, subEventType, and subEventKey.

Challenge: If the same test user is used across multiple tests, or if manual tests are run concurrently, duplicate events for the same user may occur, leading to inaccurate results.

Solution: Use unique identifiers from the API response for identifying each test user (e.g., phone number or email or clevertap id). Avoid reusing the same test user across different tests concurrently.

5. Account for network latency before recording end time

Challenge: Event data may be delayed due to network latency or processing time.

Solution: Add a small buffer (e.g., ~5 seconds) after the functional test steps before stopping the end timestamp to allow enough time for events to be processed and generated by the system.

6. Loop through the list of events in the data table and call the Cursor API for each eventType.

7. Use the cursor obtained to retrieve the event data via the GET Events API.

8. Sanitize the JSON response by removing all control characters.

9. Linking disjoint event and event property data

Challenge: Events and their associated properties are stored separately in the API response, making it harder to correlate them.
Solution: After receiving the API response, traverse from the root records node and link the events and properties using the user profile node to ensure correct mapping of event data.

10. Within each matching profile node, look for the specified eventType and subEventType.

11. Traverse through the profiles to locate the one with the most recent last_seen timestamp.

12. Validate that the event properties align with the expected event details.

13. Ensure that the last_seen timestamp is after the recorded test execution start time and within the test execution end time. The last_seen attribute is used to identify and validate the most recent pair of profile:event name within the test execution window.

14. Mark the result as "found" if the event exists in the cursor; otherwise, mark it as "not found".

15. If the event is found, pass the test for that specific event.

16.  Getting the next cursor

Challenge: Long-running tests or frequent event triggers may cause events to be excluded from the initial cursor response due to batch size limitations.

Solution: If the event is not found, check for next_cursor (if available) in the API response and repeat above steps iteratively.

16.  If all cursors are exhausted without finding the event, fail the test for that event.

17.  Continue this process for each event in the data table until all are verified.

Impact of Automation

By automating analytics validation, Halodoc achieved:

• 95% Time Savings: Event validation reduced from 2 minutes per event to 5 seconds.
• Improved Reliability: Minimized the risk of missed events and incorrect data tracking.
• Eliminated manual functional testing for event validation: Since the analytics framework is integrated with existing functional suite, there is no need to perform the manual steps to trigger the event.
• Scalability: Set to run automatically on a periodic interval on Jenkins which has enabled automation of hundreds of events across multiple functional flows.

Conclusion

Automating analytics event validation offers significant benefits, including speed, accuracy, and scalability. Triggering it periodically empowers early detection of regressions or integration issues, enhancing the overall robustness of analytics tracking. While CleverTap served as our case study, the principles outlined here can be adapted to any analytics tool. By integrating event validation into functional automation, teams can ensure reliable event tracking and make informed, data-driven decisions.

Reference

Event APIs

Events represent the actions users take with your product. Examples are standard events like app launches and custom events you define like product views. 📘 Events Used for Live Campaigns: Any events used for live (triggered) campaigns must be in real-time or as close as possible (i.e., 2 hours in …

CleverTap Developer Docs

Join us

Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.

About Halodoc

Halodoc is the number 1 all round Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. We recently closed our Series D round and in total have raised around USD $100+ million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.