Migrating Halodoc Search from Algolia to Elastic Search

Our mission at Halodoc of constantly simplifying patient experience often leads us to exploring various cutting edge technologies. So subsequently, one of the core components we wanted to improve was our Search.

Search is a ubiquitous presence across our products. From our medicine delivery all the way to the hospital appointment services, search plays a significant role in helping our users. Until recently, all of our search flows were powered by Algolia but we have since migrated to Elastic Search. This blog briefly talks about why we decided to move away from Algolia, the challenges we faced during migration and how we overcame those to deliver an improved Halodoc to our users.

Algolia overview

Algolia is a hosted search engine that offers out of the box search functionalities like full text search, fuzzy search, faceting, geospatial search among others. With a very minimal configuration and simple SDK to integrate, It worked very well for our search use cases without a lot of overhead.

At Halodoc, we used Algolia for our search use cases across multiple business verticals such as:

• Medicines Search for Pharmacy delivery
• Doctors Search for Tele-consultation

Why did we decide to move away from Algolia?

After a closer look at our data, we noticed that our search volume was skewed towards queries related to new products. These require sensitive information like deeper user-related and catalog data to produce a result.

1. Algolia is a hosted search engine which meant that our data was hosted on their servers. Moving forward, we want to host all our data within our own in-house VPC and implement in-house search solution
2. Algolia pricing is based on data size and the number of read/write operations.
   As we scale, this becomes very expensive
3. To have the flexibility to scale up/down with our search infrastructure

Having this clarity on our requirements, we decided to switch from Algolia to Elastic Search.

Migration Challenge: Matching Algolia Relevancy

Algolia covered almost all of our Pharmacy Delivery use and it gave us very relevant results. The biggest challenge we faced was matching Algolia Relevancy to our medicines search results.

Implementation: Matching Algolia Relevancy In Elastic Search

Searching for medicines is enabled primarily on two fields: name and meta-keywords. This is the sample structure of our medicine record.

"name": "PARACETAMOL DROPS",
"meta_keywords": "panadol, naprex, ottopan, pamol"

In General, Elastic Search has two steps

• Indexing data - Insert the data (slightly different from traditional databases as data is stored in an Inverted Index)
• Searching data - Retrieve the data using queries

In this blog we will be exploring more about data retrieval part and how we are able to match Algolia relevancy using Elastic Search.

As a first step, we took the top 300 search queries and analysed the patterns of search queries and corresponding results from Algolia.

Following are few of frequent query patterns that we observed and how we implemented them in Elastic Search

1. Search with Medicine Prefix name

• When users search with first few characters of medicine name, we need to show all the medicines that begin with those characters.
• Ex: If a user searches parac we should be able to match paracetamol, paraco sirup etc..

• Elastic Search Implementation: For this, we used a Prefix Query on our search fields.

2. Search with Exact Medicine name

• Ex: If a user searches paracetamol, we should be able to match all the medicines containing paracetamol in the name or meta keywords field.

• Elastic Search Implementation: For this, we used Match Query, Match Phrase Query on our search fields.

3. Searches with typos in medicine name

• Ex: If user searches paracetmal, we should be able to match it with the paracetamol by overlooking the typo

• Elastic Search Implementation: For this we used Fuzzy Query on our search fields

4. Medicine Search with language specific words

• Ex: A user searches Obat batuk cap ibu dan anak ( English Translation: Cough medicine for mother and child). This is one of the most frequently searched terms in Bahasa (Indonesian Language).

• Elastic Search Implementation: As our users are from Indonesia we used Indonesian Stop Token Filter to ignore the most frequent language specific words (ignore words like and, the, a, an , etc… equivalent english words in Bahasa)

5. Medicine Search with variant

• Sample Queries: Panadol 15 ml, Panadol 30 ml, Panadol30ml (without spaces), etc..

• Elastic Search Implementation: Identifying variant in search queries without spaces is tricky as we can’t derive the variant from single word. This made us explore WordDelimiter Token Filter further in Elastic Search.
• Ex: if query is - "Panadol30ml" WordDelimiter Token Filter will split it into 3 words [‘panadol’, ’30’, ‘ml']

After applying WordDelimiter token filter on search queries, we now are successfully able to identify the medicine name and variant parts in our search query. Furthermore, we used Match Query, Match Phrase Query  on search fields thereafter to match medicine name and variant .

Evaluation: Algolia Vs Elastic Search Search Results

• Testing Search Relevancy is not similar to testing any other functional feature. For Search relevancy, along with evaluating the number of results and pagination, we need to also look at the efficacy, order and fuzziness of the results and how the relevancy of results change with additional data.
• We developed an internal tool that helps us compare the above metrics for a given search query. After evaluating the search results for our top 600 queries, we migrated our production traffic to Elastic Search.

Summary

Overall, we are able to match our search relevancy with Algolia in almost all of our use cases except the combination of  'Prefix and Fuzzy' search queries. While this didn't have an immediate effect on our order conversions, we were able to address the privacy and scalability issues at play.

We are always looking out to hire for all roles in our tech team. If challenging problems that drive big impact enthral you, do reach out to us at careers.india@halodoc.com

About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke.
We connect 20,000+ doctors with patients in need through our teleconsultation service, we partner with 1500+ pharmacies in 50 cities to bring medicine to your doorstep, we partner with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allows patients to book a doctor appointment inside our application.
We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates foundation, Singtel, UOB Ventures, Allianz, Gojek and many more. We recently closed our Series B round and In total have raised USD$100million for our mission.
Our team work tirelessly to make sure that we create the best healthcare solution personalized for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.