CoinDCX Effortlessly Built Automated API Based ETL Pipelines and Streamlined Data Management on AWS

Introduction

CoinDCX, India’s most valuable crypto investment app, dedicated to make crypto accessible in a simple way. Established in 2018, CoinDCX has solved numerous problems faced by the Indian crypto community with solutions around crypto investing, crypto trading & crypto literacy.

Challenges

With the existing set-up CoinDCX were facing data security issues and the process was time consuming with manual data management. It was error prone and led to inaccuracies in the data, which could impact the overall quality and reliability of the data, so they wanted a data platform for a single source of truth of data to solve data integrity, security issues and ETL pipelines for automated data processing through API Data Engineering to ingest and curate the data from various third party source systems.

Searce Solution

CoinDCX’s data source was Third Party Application and CSV files. Data would be pulled from Third Party Vendor Application through API and CSV files to S3 using MWAA and Custom DAG. Searce solution used the MWAA and created a custom DAG to pull data from APIs and CSV to S3 and schedule the pipeline to run and pull the data from source & merge the data as per requirement. Created ETL pipelines for data transformation using spark on Databricks as per the CoinDCX requirement to read the data from S3 and curate it. Transformed & curated data stored in Data Lake (S3). Also, Searce has developed data analytics applications in Databricks for the OLAP data which would be consumed by BI applications for business reports.

Business Impact

With the new Architecture design and AWS’ services CoinDCX was able to:

Improved Workflow Automation and data management, MWAA helped CoinDCX to automate their workflows and processes, which can result in improved efficiency, reduced manual effort, and increased accuracy
By automating workflows and reducing manual effort, MWAA and S3 helped CoinDCX to save time and cost on data processing
The Implemented architecture helped CoinDCX to avoid the overhead of managing their own infrastructure, and can help ensure that workflows continue to scale and run smoothly, even during periods of high demand
Data Lake architecture for fine grained access control and single source of truth for data.

Searce’s solution not only enabled CoinDCX to effortlessly create a fortified API Data Engineering leveraging the cutting-edge AWS Services – MWAA and S3, but also facilitated seamless data management and data processing from multiple third party source systems. By automating workflows and storing data in S3, our solution effectively expedited data processing, reduced redundancy, and streamlined data management, thereby ensuring the utmost security of the data analytics platform. With our solution, CoinDCX was able to optimize their workflows and leverage business intelligence dashboards to gain valuable insights and make swift, well-informed decisions.

Industry: Finance
Workload: AWS, AWS Networking Components, Amazon Managed Apache Airflow, Amazon S3, Amazon Cloudwatch