Migrating from ElasticSearch 6.8 to OpenSearch 1.3
Migrating from ElasticSearch 6.8 to OpenSearch 1.3 can be a complex process, especially when dealing with a legacy system that functions as an analytics layer. In this blog post, I’ll share my recent experience of upgrading a system from ElasticSearch 6.8 to AWS OpenSearch Service 1.3, addressing some of the challenges we faced and how we overcame them.
Background: The Legacy System Setup
The legacy system we were working on is designed to process data from various sources and serves as an analytics layer. Here’s a brief overview of the setup:Data Ingestion and Processing: Data is ingested from multiple sources, stored in Oracle, and processed using an EMR cluster via Spark. After the ETL process, the data is loaded into ElasticSearch and indexed.Infrastructure and Deployment: The system employs a daily blue-green deployment strategy. Every day, new EMR and ElasticSearch clusters are built using a CloudFormation template as soon as the jobs are initiated. Once the new cluster is successfully built and loaded, the DNS is switched to the new cluster, and the old cluster is deleted.Given this daily cluster rebuilding process, we assumed that updating the CloudFormation template and running our jobs would suffice for the migration. However, we encountered an issue where data failed to load during this process, and the logs provided minimal information.
The Migration Challenge: Bulk Operation Errors
During the migration, we followed AWS documentation closely but still ran into issues, specifically a bulk operation error:"Could not write all entries for bulk operation [1119/1119]."This error was directly related to the index templates and mapping types used in our setup.
The Solution: Step-by-Step Resolution
After some troubleshooting, we were able to resolve the issues by taking the following steps:
Update the CloudFormation Template:The first step was to update the CloudFormation template to reflect the OpenSearch 1.3 domain configuration. This involved modifying various parameters to ensure compatibility with OpenSearch.
Address Bulk Operation Errors:After making the template updates, we encountered the bulk operation error mentioned above. This issue was due to changes in how OpenSearch handles index templates and mapping types.
Index Template Changes:We updated our index templates where the mapping type was previously set to "person" to "_doc" as required by OpenSearch 1.x. This change is crucial because OpenSearch no longer supports custom mapping types, and "_doc" is the default type.
Handle Malformed Data:To handle any potential issues with malformed data, we added the setting `"index.mapping.ignore_malformed": true` to the index. This allowed the system to gracefully ignore any malformed data rather than failing outright.
Include Type Name for Compatibility:We also set the `include_type_name=true` condition when creating the index. This was necessary to ensure compatibility with our existing data structure, which relied on type names.
Results: Successful Migration and Smooth Operations
After implementing these changes, the migration was successful. Our data now loads correctly, and the clusters operate smoothly in the new OpenSearch environment. This experience highlights the importance of understanding the nuances of OpenSearch, especially when migrating from an older version of ElasticSearch.
Key Takeaways
Plan Ahead: Migration from ElasticSearch to OpenSearch requires careful planning, especially regarding index templates and mapping types.
Test Thoroughly: Always test your configuration changes in a staging environment before applying them to production.
Be Prepared for Errors: Bulk operation errors can be tricky, but understanding how OpenSearch handles index templates and mappings can help resolve these issues.I hope this guide helps others who are navigating the migration process from ElasticSearch 6.8 to OpenSearch 1.3.
If you encounter similar issues, these steps should guide you toward a smooth transition.
