Ensuring Data Consistency for Customers Using Snapshots Concept

In the world of data management and system design, ensuring data consistency is paramount. One crucial technique to achieve this is snapshotting. In this blog post, we'll explore the concept of snapshotting and how it is used to maintain data consistency, using a practical scenario to illustrate its importance.

Overview

Emphasizes the criticality of data consistency within systems focused on customer data management. It specifically highlights Catalog Level Snapshotting as a key method to uphold this consistency, comparing its advantages to the conventional Transaction Level Snapshot approach. Snapshotting, at its core, serves as a foundational technique to maintain data integrity by capturing and preserving specific data states at distinct moments. This functionality is essential as it enables efficient restoration of these historical data points when necessary. By allowing systems to reference or revert to snapshots, this method provides a comprehensive view of data at precise instances despite ongoing changes. Consequently, it forms a reliable historical record pivotal for efficient data management and robust system recovery strategies.

The Issue in Absence of Snapshots

In the absence of Snapshots or a mechanism to capture and retain specific states of data at given points in time, several challenges and issues can arise:

1. Data Integrity and Consistency: Without snapshots, it becomes challenging to maintain the integrity and consistency of data, especially in systems where frequent changes occur. Updates, deletions, or modifications to data entries can lead to a loss of historical context, making it difficult to track changes or revert to previous versions if needed.
2. Inaccurate Historical Records: The absence of snapshots results in a lack of historical records or a clear audit trail of data changes. This can be critical in various industries, especially healthcare or finance, where accurate historical data is essential for compliance, audits, and decision-making.
3. Difficulty in Resolving Issues: Troubleshooting and resolving issues related to data discrepancies or unexpected outcomes become more challenging without snapshots. It's harder to pinpoint when and where a change occurred, making it time-consuming to identify and rectify issues.
4. Impact on Customer Experience: In scenarios such as service-based platforms, the absence of snapshots could affect customer experience. Customers might receive unexpected changes in products or services without the ability to trace back to the version they initially interacted with, leading to confusion or dissatisfaction.
5. Compliance and Regulatory Concerns: In industries governed by strict regulations, such as healthcare or finance, the lack of proper data versioning through snapshots can pose compliance risks. Auditors or regulators might require detailed historical records, which the system might not adequately provide.
6. Increased Risk in Data Management: Without a snapshot mechanism, the risk of data loss or inadvertent data alterations increases. Accidental deletions or modifications without a way to restore previous versions can lead to irreversible consequences.

Before we use Snapshots let's examine a practical scenario to understand the concept better. Customer A placed an order for "Product A" when it was at version 1. The key here is that we want to ensure that Customer A always sees the product they ordered, regardless of any changes made to the product later.

The beauty of snapshotting in this scenario is that when Customer A placed their order for "Product A" version 1, the system took a snapshot of that specific product's data. This means that even if "Product A" gets updated to version 2 with a different price, Customer A's order remains associated with the original version, and they get the price they agreed to when they placed their order.

In the ever-evolving landscape of data management, the strategic use of snapshots becomes imperative for maintaining data integrity and facilitating efficient system operations. This blog delves into two distinct approaches to snapshotting within our system: Transaction Level Snapshot and Catalog Level Snapshot. Each approach offers unique benefits, addressing scalability challenges while ensuring data consistency in different scenarios.

Transaction Level Snapshot

The Transaction Level Snapshot approach involves capturing snapshots against each order, ensuring precise data representation at the point of purchase. While effective in maintaining data accuracy, the Transaction Level Snapshot approach excels in providing an exact and detailed representation of data at the moment of each transaction. This precision ensures users receive accurate information at the time of purchase, aiding in trust-building and precise transaction records.

Storing snapshots against each order, while a simpler approach, may become less scalable and present challenges as the system grows or deals with a substantial volume of data. Here's an explanation:

1. Increased Storage Requirements: As the number of orders increases, storing snapshots against each order leads to a significant increase in data storage requirements. This approach can become resource-intensive, especially when dealing with large volumes of orders or frequent changes to data.
2. Complexity in Retrieval and Management with Performance Implications: Storing snapshots against each order not only increases the complexity in retrieval and management, especially with a growing number of orders, but also raises concerns regarding system performance. Accessing and managing individual snapshots for each order becomes increasingly cumbersome and less efficient as the volume of orders grows. Moreover, this method might impact system performance by potentially slowing down data retrieval processes, resulting in latency issues or reduced system responsiveness.
3. Difficulty in Tracking Changes: As the system scales, tracking changes and managing snapshots at the order level might become challenging. Ensuring consistency and accuracy in associating the right snapshot with the corresponding order might become more complex with a larger dataset.
4. Scalability Concerns: The approach of storing snapshots against each order might face scalability limitations. As the system grows, scalability concerns may arise due to the increasing volume of data and the system's ability to efficiently handle and manage these snapshots.

Transaction Level Snapshotting provides accurate and granular data representation at each transaction, aiding in detailed record-keeping and compliance. However, it poses challenges in scalability due to increased storage needs and retrieval complexities. Performance issues and difficulties in maintaining snapshot consistency also arise as the system scales, potentially impacting data accuracy and system responsiveness.

Catalog Level Snapshot

The Catalog Level Snapshot introduces an innovative approach to snapshot management, where identical product versions across different orders share a common Snapshot ID. This unique method optimizes storage efficiency and enhances data retrieval processes. This section explores the strategic advantages of this unified approach, emphasizing its ability to streamline storage, simplify retrieval tasks, and promote system-wide coherence by referencing a single Snapshot ID for identical product versions across multiple orders. The Catalog Level Snapshot method signifies a paradigm shift toward an optimized and cohesive snapshot system, ensuring resource efficiency and unified data representation in evolving system environments.

Implementing version control for data entries involves creating a structured framework that tracks and manages different iterations or versions of data. In the scenario of the Halodoc Platform, this means efficiently handling various versions of products.

1. Free Audit Trails and Compliance Records: Snapshots create detailed audit trails, facilitating compliance with regulatory standards. Historical snapshots serve as records showcasing the system's compliance efforts over time, aiding in regulatory audits and demonstrating adherence to industry standards.
2. Database Versioning: Utilize database features or implement a versioning system that allows tracking changes to individual data records. For instance, each product entry should have a versioning system integrated into the database schema.
3. Version Identification: Assign a unique identifier or timestamp to each version of a data entry. This identifier helps in differentiating and referencing specific versions of products. It could be a sequential number, a timestamp, or a combination of both.
4. Data Retrieval Mechanism: Create retrieval mechanisms that enable accessing specific versions of data. This could involve custom queries or APIs that retrieve the appropriate version based on the order history and associated snapshots.
5. Update and Rollback Procedures: Establish procedures for updating data entries while maintaining previous versions. This includes protocols for rolling back to previous versions if required, ensuring that customer orders align with the correct product versions at the time of purchase.
6. Integration with Order Management System: Integrate version control with the order management system so that each order is linked explicitly to the version of the product available at the time of purchase.
7. Testing and Validation: Thoroughly test the version control mechanisms to ensure accuracy, reliability, and scalability. Validate that data retrieval, updates, and rollbacks occur seamlessly without compromising system performance.

In Halodoc we consider employing the Catalog Level Snapshot method due to the necessity of managing a diverse array of health-related orders and numerous product versions within our health tech ecosystem.

This approach efficiently handles a large volume of orders while maintaining streamlined data representation.

The Catalog Level Snapshotting optimizes storage by consolidating identical product versions across various orders under a unified Snapshot ID. This consolidation reduces redundancy and conserves resources. By referencing a single Snapshot ID for similar product versions, data retrieval processes are streamlined, simplifying access and management across multiple orders. This method aligns well with Halodoc's needs, ensuring efficient management of diverse product versions and scalability within our health tech platform.

Conclusion

In Halodoc's Halolab product, the focused utilization of snapshots is specifically tailored to manage Test and Lab Packages offered through our app. These snapshots play a critical role in maintaining the accuracy and consistency of data associated with these packages. By employing Catalog Level Snapshots, Halodoc ensures that the details and specifications of Test and Lab Packages remain precise and unaltered, safeguarding the integrity of the information available to users.

Moreover, the strategic use of snapshots simplifies the customer experience by guaranteeing that individuals receive the exact version of the Test or Lab Package they selected, regardless of any subsequent modifications made to these packages. This approach ensures a consistent and reliable healthcare service, aligning with customer expectations and needs. Additionally, the availability of snapshots facilitates easy reference to historical package details, aiding healthcare professionals and internal management in analyses and comparisons for improved decision-making and service optimization within Halodoc's health tech ecosystem.

References

• https://medium.com/@mweststrate/distributing-state-changes-using-snapshots-patches-and-actions-part-1-2811a2fcd65f
• https://neptune.ai/blog/version-control

Join Us

Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resume at careers.india@halodoc.com.

About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. We recently closed our Series D round and in total have raised around USD$100+ million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.