NewRelic custom integration for MySQL RDS

NewRelic Jun 27, 2025

Introduction

In the fast-paced world of healthcare technology, particularly at Halodoc, database performance is paramount. Slow queries can significantly degrade application responsiveness, leading to frustrated users, in our domain, potentially impacting critical services. Traditionally, monitoring these elusive performance bottlenecks often involved sifting through verbose CloudWatch logs—a process that can be both time-consuming and costly.

By centralizing our slow query data in New Relic, Halodoc aimed to gain unprecedented visibility into our database performance, enabling our teams to quickly identify and address performance bottlenecks—ultimately leading to a more robust and responsive system for our users.

To support this, we made a strategic decision to move away from AWS CloudWatch for slow query monitoring and adopt New Relic. This migration was driven by the need for significant cost reduction (approximately 95% savings over cloudwatch logs), structured and easily parsed logs, and simplified observability tailored to teams and applications.

Why Move Away from CloudWatch?

While AWS CloudWatch is a robust monitoring service, it has several limitations when it comes to slow query monitoring:

  1. Cost Considerations: CloudWatch’s pay-per-use model leads to high costs due to log ingestion, storage, limited retention, and frequent dashboard queries, especially in high-volume database environments.
  2. Limited Query Analysis: CloudWatch offers only basic alerting and visualization, lacks AI-driven insights, and provides minimal support for in-depth query performance analysis or optimization.
  3. Operational Challenges: CloudWatch requires complex setup for log parsing, lacks seamless correlation between slow queries and application metrics, and results in fragmented monitoring across systems.

Features of New Relic Infrastructure Agent

New Relic offers several advantages for database monitoring:

  1. Cost Optimization: Provides predictable, fixed pricing at approximately $0.35/GB based on ingestion volume, along with efficient data compression that reduces storage costs. It also offers a unified platform for monitoring infrastructure, applications, and databases. For a medium-scale enterprise with 80–100 MySQL RDS instances, the total ingested log volume typically amounts to around 1.3 GB per month, making it a highly cost-effective solution.
  2. Enhanced Visualization: Enables rich, customizable dashboards with a drag-and-drop interface, supports team-based access control for tailored views, and integrates advanced alerting with multi-channel notifications.
  3. Integration Capabilities: Enables direct correlation between slow queries and application performance, provides a holistic view across infrastructure and applications, and supports seamless integration with popular DevOps tools.

Implementation Details

The entire implementation was broken down into 3 stages so that we can get optimal results.

  1. Pre-Implementation Stage: This stage involves steps to ensured access to RDS, New Relic, and EKS, and preparing all the required credentials. We also included monitoring scope, team-specific dashboards designs, and plan out the the schedule necessary for implementing the RDS configuration changes.
  2. Implementation Stage: In this stage, we enabled slow query logging on each of the RDS instances and deployed the New Relic agent using Helm with a secure, custom-configured values.yaml. We also set up encrypted secrets and synthetic monitoring for enhanced visibility.
  3. Post-Implementation Stage: In this stage, we have validated data ingestion, alert accuracy, and dashboard functionality. We are also optimizing performance by fine-tuning collection settings and configuring dashboards for long-term observability.
Process Flow

Pre-Implementation Stage

This stage involves the following steps:

  1. Access Requirements: Ensure AWS RDS administrative access, a New Relic account with the appropriate licensing, and access to the EKS cluster for agent installation.
  2. Planning Considerations: Identify all RDS instances that need monitoring, define team-specific log groupings, determine dashboard requirements for each team, and schedule maintenance windows for necessary RDS parameter changes.
  3. Prerequisites: Ensure MySQL RDS instances are running and accessible, verify network connectivity between the EKS cluster and RDS, and have the New Relic license key available.

Implementation Stage

We will start by enabling slow query logging on RDS by modifying key parameters and rebooting the instance. Deploy the New Relic Infrastructure agent using Helm for repeatable, version-controlled setup, supported by a secure and encrypted credentials strategy. Add the New Relic Helm repository, create a tailored values.yaml for your RDS instances, and deploy the bundle. Advanced configuration allows scalable integration using secrets, custom attributes, and forwarding-only mode. Implement synthetic monitoring to enhance historical slow query visibility.

Configure MySQL Slow Query Logging
To configure slow queries in RDS, we have to modify the RDS parameter group by setting slow_query_log = 1, long_query_time = 1, and log_output = TABLE to enable slow query logging. After making these changes, navigate to the AWS RDS Console, select your MySQL instance, and apply the modified parameter group. A reboot of the RDS instance is required for the changes to take effect.
These commands will confirm that slow query logging is enabled, the correct threshold is set, and that slow queries are being recorded in the mysql.slow_log table.
Once the instance has restarted, verify the configuration using the following SQL commands:

Configuration Verification

Install New Relic Infrastructure Agent
For Kubernetes environments, deploying New Relic with Helm charts offers a streamlined, repeatable, and configurable installation process. Helm simplifies the deployment of complex applications by managing all resources through version-controlled templates, making it easy to customize settings like license keys, namespaces, and MySQL slow query integrations via a values.yaml file. This approach ensures consistency across environments and enables faster rollouts and upgrades.

  1. Encrypting DB username and password
    We do not store the database username and password in plain text within values.yaml. Instead, the password is encrypted using New Relic-supported encryption and then obfuscated before being stored as a Kubernetes Secret. This approach ensures the credentials remain highly secure. We have used below method to encrypt the password -
Credentials Encryption
  1. Add New Relic Helm Repository
    To install New Relic components via Helm, first add the official New Relic Helm chart repository and update your local chart index to ensure access to the latest versions.
Helm Add Repo
  1. Create Values File
    Define a custom values.yaml file to configure the New Relic Infrastructure agent and Flex integration for MySQL slow query monitoring. This file includes EKS namespace settings, license key and secret references, resource limits, and detailed MySQL query configurations for multiple RDS endpoints.
Namespace: newrelic licenseKey: "" cluster: "" clusternamehack: '{"clusterName": ""}' customSecretName: "" customSecretLicenseKey: "" nameOverride: "" fullnameOverride: "" image: registry: repository: newrelic/infrastructure-bundle pullPolicy: IfNotPresent tag: "3.2.24" imagePullSecrets: [] config: nri-mysql.yaml: | integrations: - name: nri-flex interval: 300s config: name: MySQLDbFlex secrets: mylogin: kind: local key: {{DB_KEY_ENV_VARIABLE}} data: {{DB_DATA_KEY_ENV_VARIABLE}} type: equal apis: - database: mysql db_conn: "${secret.mylogin:USER_NAME}:${secret.mylogin:PASS_WORD}@tcp(RDS_ENDPOINT_1:PORT)/sys" logging: open: true custom_attributes: RDS_HOST: RDS_HOST_1 db_async: true db_queries: - name: Newrelic_group run: select start_time, user_host, TIME_TO_SEC(query_time)+(MICROSECOND(query_time)/1000000) as query_time_sec, lock_time, rows_sent, rows_examined, db, last_insert_id, insert_id, server_id, sql_text, thread_id from mysql.slow_log where start_time >= TIMESTAMPADD(MINUTE, -5, CURRENT_TIMESTAMP); custom_attributes: attype: slow-query host: DB_1 - database: mysql db_conn: "${secret.mylogin:USER_NAME}:${secret.mylogin:PASS_WORD}@tcp(RDS_ENDPOINT_2:PORT)/sys" logging: open: true custom_attributes: RDS_HOST: RDS_HOST_2 db_async: true db_queries: - name: Newrelic_group run: select start_time, user_host, TIME_TO_SEC(query_time)+(MICROSECOND(query_time)/1000000) as query_time_sec, lock_time, rows_sent, rows_examined, db, last_insert_id, insert_id, server_id, sql_text, thread_id from mysql.slow_log where start_time >= TIMESTAMPADD(MINUTE, -5, CURRENT_TIMESTAMP); custom_attributes: attype: slow-query host: DB_2 serviceAccount: create: annotations: name: deploymentAnnotations: {} podAnnotations: {} labels: {} podLabels: {} privileged: priorityClassName: "" hostNetwork: podSecurityContext: {} containerSecurityContext: {} dnsconfig: {} resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi nodeSelector: {} tolerations: [] affinity: {} proxy: lowDataMode: nrStaging: fedRamp: enabled: verboseLog:
  1. Deploy Using Helm
    Use Helm to install the New Relic Infrastructure bundle with your custom values.yaml for MySQL slow query monitoring. Post-deployment, verify that the pods are running and logs indicate successful agent initialization.
Helm Installation
  1. Deployment for Advanced Configuration
    This Kubernetes Deployment sets up the New Relic Infrastructure agent in an advanced configuration mode within an EKS cluster. It uses secure environment variables, secret-based credentials, and a mounted config map to enable fine-grained control, forwarding-only telemetry, and scalable integration management.
apiVersion: apps/v1 kind: Deployment metadata: labels: {{- include "newrelic.common.labels" . | nindent 4 }} annotations: {{- if .Values.deploymentAnnotations }} {{- toYaml .Values.deploymentAnnotations | nindent 4 }} {{- end }} name: {{ include "newrelic.common.naming.fullname" . }} namespace: {{ .Values.Namespace }} spec: replicas: 1 selector: matchLabels: app: newrelic-infra-ohi template: metadata: {{- if .Values.podAnnotations }} annotations: {{- toYaml .Values.podAnnotations | nindent 8}} {{- end }} labels: app: newrelic-infra-ohi {{- include "newrelic.common.labels.podLabels" . | nindent 8 }} spec: securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }} containers: - name: newrelic-infra securityContext: {{- toYaml .Values.containerSecurityContext | nindent 12 }} image: newrelic/infrastructure-bundle:latest volumeMounts: - name: config-volume mountPath: /etc/newrelic-infra/integrations.d env: - name: NRIA_IS_FORWARD_ONLY value: "true" # - name: NRIA_IS_SECURE_FORWARD_ONLY # value: "true" - name: NRIA_LOG_LEVEL value: "info" - name: "NRIA_OVERRIDE_HOSTNAME_SHORT" valueFrom: fieldRef: apiVersion: "v1" fieldPath: "spec.nodeName" - name: "NRIA_OVERRIDE_HOSTNAME" valueFrom: fieldRef: apiVersion: "v1" fieldPath: "spec.nodeName" - name: NRIA_CUSTOM_ATTRIBUTES value: {{ .Values.clusternamehack | squote }} - name: NRIA_LICENSE_KEY valueFrom: secretKeyRef: name: newrelic-key-env-variable key: NEWRELIC_KEY_ENV_VARIABLE - name: DB_KEY valueFrom: secretKeyRef: name: db-key-env-variable key: DB_KEY_ENV_VARIABLE - name: DB_KEY_DATA valueFrom: secretKeyRef: name: db-key-data-env-variable key: DB_KEY_DATA_ENV_VARIABLE resources: {{- toYaml $.Values.resources | nindent 12 }} volumes: - name: config-volume configMap: name: {{ include "newrelic.common.naming.fullname" . }}
  1. Create synthetic monitoring
    To enable consistent visibility into slow queries across each service on a weekly basis, we’ve implemented synthetic monitoring that retrieves data from the primary slow query log group every week. This setup helps us overcome the limitations of NRQL when dealing with historical or grouped data. The collected data is then pushed into a separate, structured log group, allowing for better aggregation and analysis. As a result, we now have a more refined and focused view of our P0 (high-priority) queries, improving our ability to troubleshoot and optimize database performance.
var assert = require('assert'); const myAccountId = '1234567'; const myAPIKey = 'NRI_API_KEY'; const NR_LICENSE_KEY = 'LICENSE_KEY'; const defaultOptions = { uri: 'https://api.newrelic.com/graphql', headers: { 'API-key': myAPIKey, 'Content-Type': 'application/json', } }; const EVENT_URL = 'https://insights-collector.newrelic.com/v1/accounts/1234567/events'; const headers2 = { "Content-Type": "application/json", "Api-Key": NR_LICENSE_KEY, }; const options = { uri: defaultOptions.uri, headers: defaultOptions.headers, body: JSON.stringify({ query: ` query getNrqlResults($accountId: Int!, $nrql: Nrql!) { actor { account(id: $accountId) { nrql(query: $nrql) { results } } } } `, variables: { accountId: Number(myAccountId), nrql: 'SELECT count(*) AS \'EXECUTIONS\', max(query_time_sec) AS \'MAX_EXECUTION_TIME\' FROM newrelic_group FACET RDS_HOST AS \'RDS_HOSTNAME\', db AS \'DATABASE\', sql_text AS \'SQL_TEXT\', user_host AS \'USER_HOST\' WHERE user_host NOT LIKE \'user3%\' AND user_host NOT LIKE \'user1%\' and user_host not like \'user4%\' AND user_host NOT LIKE \'user2%\' AND sql_text not like \'-- sql_text%\' AND sql_text not like \'-- sql_text%\' and user_host NOT LIKE \'user5%\' SINCE 7 DAY AGO LIMIT MAX' } }), }; function callback(err, response, bodyStr) { if (err || response.statusCode !== 200) { console.error("NRQL query failed:", err || `Status ${response.statusCode}`); return; } let body; try { body = typeof bodyStr === 'string' ? JSON.parse(bodyStr) : bodyStr; } catch (e) { console.error("Failed to parse response JSON", e); return; } const results = body.data.actor.account.nrql.results; if (!results || results.length === 0) { console.warn("No results found."); return; } const filteredResults = results.filter(r => r.EXECUTIONS > 1000 || r.MAX_EXECUTION_TIME > 10 ); console.log(`Filtered ${filteredResults.length} records for posting.`); const BATCH_SIZE = 5; const DELAY_BETWEEN_BATCHES = 1000; let index = 0; function processBatch() { const batch = filteredResults.slice(index, index + BATCH_SIZE); if (batch.length === 0) { console.log("All batches processed."); assert.ok(true, 'Script completed successfully.'); return; } console.log(`Processing batch ${index / BATCH_SIZE + 1} (${batch.length} entries)`); let completed = 0; batch.forEach((entry, i) => { $http.post({ url: EVENT_URL, headers: headers2, body: JSON.stringify({ eventType: "new_log_group", RDS: entry.facet[0], DB: entry.facet[1], SQL_TEXT: entry.facet[2], USER_HOST: entry.facet[3], EXECUTIONS: entry.EXECUTIONS, MAX_EXECUTION_TIME: entry.MAX_EXECUTION_TIME, }), }, (error, response, body) => { if (error || response.statusCode !== 200) { console.error(`POST failed for index ${index + i}`); console.error(`Status: ${response?.statusCode}, Error:`, error || body); } else { console.log(`POST succeeded for index ${index + i}`); } completed++; if (completed === batch.length) { index += BATCH_SIZE; setTimeout(processBatch, DELAY_BETWEEN_BATCHES); } }); }); } processBatch(); } $http.post(options, callback);

Post-Implementation Stage

This stage involves the following steps to verify data ingestion, validate alerts and dashboards, and optimize monitoring through tuned settings and customized visualizations for effective slow query analysis.

  1. Verification and Testing
    Verify data collection by checking that slow query logs are being captured in New Relic, ensuring data is consistently ingested from RDS and accurately reflects query performance. Next, test the alert configurations by creating controlled scenarios that generate slow queries to ensure alerts are triggered appropriately. Then, validate the dashboard functionality by confirming that all widgets display the expected data accurately and update in real-time, reflecting ongoing query performance metrics.
    Enhance performance, by adjusting the data collection intervals to suit your system’s query volume and performance requirements. Then, optimize query filters to focus on the most critical slow queries, helping reduce noise and improve clarity. Lastly, configure appropriate data retention policies to balance the need for historical analysis with storage cost efficiency.
  1. Dashboard Configuration
    The following New Relic dashboard JSON provides comprehensive visualization of your MySQL slow query data:
{ "name": "MySQL Slow Queries Dashboard Prod", "description": null, "permissions": "PUBLIC_READ_ONLY", "pages": [ { "guid": "test_guid", "name": "MySQL Slow Queries Dashboard", "description": null, "widgets": [ { "id": "1", "title": "RDS Slow Query Analysis", "layout": { "column": 1, "row": 1, "width": 6, "height": 3 }, "linkedEntityGuids": ["test_guid"], "visualization": { "id": "viz.bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountIds": [1234567], "query": "select count(*) from Newrelic_group where user_host NOT LIKE 'USER%' AND user_host NOT LIKE 'USER%' AND user_host NOT LIKE 'USER%' AND sql_text not like '-- QUERY_PARAM%' AND sql_text not like '-- QUERY_PARAM%' AND RDS_HOST IN ({{RDS_HOST}}) facet RDS_HOST limit max" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "id": "2", "title": "Daily Slow Query Count", "layout": { "column": 7, "row": 1, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "markers": { "displayedTypes": { "criticalViolations": false, "deployments": true, "relatedDeployments": true, "warningViolations": false } }, "nrqlQueries": [ { "accountIds": [1234567], "query": "select count(*) from Newrelic_group where user_host NOT LIKE 'USER%' AND user_host NOT LIKE 'USER%' AND user_host NOT LIKE 'USER%' AND sql_text not like '-- QUERY_PARAM%' AND sql_text not like '-- QUERY_PARAM%' AND RDS_HOST IN ({{RDS_HOST}}) facet RDS_HOST limit max TIMESERIES" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "id": "3", "title": "Grouped Query Insights", "layout": { "column": 1, "row": 4, "width": 12, "height": 4 }, "linkedEntityGuids": ["test_guid"], "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountIds": [1234567], "query": "SELECT count(*) AS 'EXECUTIONS', average(query_time_sec) AS 'AVERAGE EXECUTION TIME' \nFROM Newrelic_group \nFACET RDS_HOST AS 'RDS HOSTNAME', db AS 'DATABASE', sql_text AS 'SQL TEXT' \nWHERE user_host NOT LIKE 'USER%' AND user_host NOT LIKE 'USER%' and user_host not like 'USER%' AND user_host NOT LIKE 'USER%' AND sql_text not like '-- QUERY_PARAM%' AND sql_text not like '-- QUERY_PARAM%' AND RDS_HOST IN ({{RDS_HOST}})\nLIMIT MAX \nEXTRAPOLATE " } ], "platformOptions": { "ignoreTimeRange": false } } }, { "id": "4", "title": "Query Execution Breakdown", "layout": { "column": 1, "row": 8, "width": 12, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountIds": [1234567], "query": "SELECT RDS_HOST as 'RDS HOST',db as DB,sql_text as 'SQL TEXT',user_host as 'USER & HOST',string(query_time_sec, precision: 4) as 'QUERY TIME', start_time as 'QUERY EXECUTION TIME' FROM Newrelic_group WHERE user_host not like 'USER%' AND user_host NOT LIKE 'USER%' AND user_host NOT LIKE 'USER%' and user_host not like 'USER%' AND sql_text not like '-- QUERY_PARAM%' AND sql_text not like '-- QUERY_PARAM%' AND RDS_HOST IN ({{RDS_HOST}}) limit max" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "id": "5", "title": "P0 Critical Queries", "layout": { "column": 1, "row": 12, "width": 12, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountIds": [1234567], "query": "select * from Newrelic_group since 1 week ago" } ], "platformOptions": { "ignoreTimeRange": false } } } ] } ], "variables": [ { "name": "RDS_HOST", "items": null, "defaultValues": [], "nrqlQuery": { "accountIds": [1234567], "query": "select uniques(RDS_HOST) from new_log_group order by RDS_HOST asc limit max" }, "options": { "ignoreTimeRange": false, "excluded": false }, "title": "", "type": "NRQL", "isMultiSelection": false, "replacementStrategy": "STRING" } ] }
Sample Dashboard View

Best Practices and Recommendations

Adopt the following best practices during your implementation in order to ensure Security, Performance & Monitoring.

  1. Security Considerations
    For security considerations, use dedicated monitoring user with the minimal permissions required to collect slow query data, avoiding the use of admin-level credentials. Ensure that New Relic agents have access only to the necessary database endpoints by enforcing strict network security rules. Additionally, enable SSL/TLS for all database connections to protect data in transit.
  2. Performance Optimization
    Implement intelligent query sampling to minimize overhead on high-traffic systems by focusing only on queries that exceed defined thresholds for duration or resource usage. Additionally, monitor the impact of the monitoring agent on system resources and adjust data collection intervals accordingly to ensure minimal performance disruption while maintaining visibility.
  3. Maintenance and Monitoring:
    Set up alerts to monitor agent connectivity and detect any data collection issues, and regularly review agent logs for errors or warnings to ensure continuous monitoring reliability. Additionally, maintain dashboards by periodically reviewing and updating queries for relevance, and archiving or removing outdated visualizations to keep insights focused and accurate.

Conclusion

Migrating from CloudWatch to New Relic for MySQL slow query monitoring provides significant advantages in terms of cost optimization, advanced analytics, and AI-powered insights. The implementation process, while requiring careful planning and configuration, results in a more powerful and flexible monitoring solution.

The key benefits realized from this migration include:

  1. Cost Reduction: More predictable pricing model that delivers better value for high-volume logging, enabling approximately 95% cost reduction compared to CloudWatch logs.
  2. Enhanced Visibility: Rich dashboards and visualizations offer enhanced visibility and deeper insights into database performance. It also provides developers with the single interface to identify slow performing queries and correlate it with specific and bad performing APIs.
  3. Scalability: Provides scalable support for expanding infrastructure and growing data volumes, ensuring consistent performance monitoring and visibility as system complexity and workload increase.

About Halodoc

Halodoc is the number one all-around healthcare application in Indonesia. Our mission is to simplify and deliver quality healthcare across Indonesia, from Sabang to Merauke. 

Since 2016, Halodoc has been improving health literacy in Indonesia by providing user-friendly healthcare communication, education, and information (KIE). In parallel, our ecosystem has expanded to offer a range of services that facilitate convenient access to healthcare, starting with Homecare by Halodoc as a preventive care feature that allows users to conduct health tests privately and securely from the comfort of their homes; My Insurance, which allows users to access the benefits of cashless outpatient services in a more seamless way; Chat with Doctor, which allows users to consult with over 20,000 licensed physicians via chat, video or voice call; and Health Store features that allow users to purchase medicines, supplements and various health products from our network of over 4,900 trusted partner pharmacies. To deliver holistic health solutions in a fully digital way, Halodoc offers Digital Clinic services including Haloskin, a trusted dermatology care platform guided by experienced dermatologists.

We are proud to be trusted by global and regional investors, including the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. With over USD 100 million raised to date, including our recent Series D, our team is committed to building the best personalized healthcare solutions — and we remain steadfast in our journey to simplify healthcare for all Indonesians.

Tags

Kshitij Soni

Recommended for you