Shiva Chandra Mouli T N

About Me

Data Engineer with 4+ years of experience designing scalable data pipelines on the Databricks Lakehouse using PySpark, Python, Azure Databricks, and SQL for batch and streaming workloads. Skilled in building and optimizing ETL workflows, improving performance, and delivering robust solutions in fast-paced environments. 9x Databricks certified, with a strong track record of translating business requirements into clean, maintainable data engineering solutions while applying industry best practices.

Contact Me Download CV

Work Experience

My professional journey in Data Engineering.

Data Engineer

Fractal Analytics

Sep 2023 – Present

Bengaluru, Karnataka

Validated, cleansed, and transformed data across all layers of Medallion Architecture with Unity Catalog in Azure Databricks.
Ensured 99.9% data quality and consistency by developing automated scripts using Pyspark to remove data inconsistencies and corrupt records.
Built digitization solution with Azure Data Factory (ADF), Databricks, and REST APIs, reducing manual effort by 80%.
Designed and executed dimensional data models in Delta Lakehouse to support analytical and reporting use cases.
Optimized complex SQL and PySpark ETL pipelines in Databricks using partitioning, bucketing, and caching, improving performance and reducing costs by 40%.
Collaborated with analytics and BI teams to expose curated datasets via Databricks SQL and Dashboards. Leveraged Databricks Genie for rapid insights.
Developed Spark Structured Streaming pipelines using Kafka as the source to ingest and process high-velocity data streams.
Delivered production-ready end-to-end Data Engineering solutions using Databricks Workflows, for scalable batch processing.
Dev-to-Prod migrations via Azure DevOps Repos/Pipelines, enforcing code reviews, version control, and governance standards.
Used Azure DevOps to log test results and maintain 100% accountability and traceability for the client.

Backend & Data Engineer

Accenture

June 2018 – June 2020

Designed and developed backend services using Python and Django, including models, migrations, serializers, and querysets, achieving 100% feature delivery.
Built and integrated RESTful API endpoints ensuring 99.9% uptime and smooth backend-frontend integration.
Developed Django views and URL routing, leveraging Django’s MVT framework and ORM for efficient data management.
Executed unit testing with Pytest, authored UI and backend test cases, tracked QA progress in Azure DevOps, and maintained over 85% test coverage.
Participated in requirement gathering and documentation, contributing technical insights that cut early-stage revisions by 15%.
Managed MySQL databases and used Azure Data Factory for artifact storage and data workflows.

Featured Projects

Key initiatives and solutions aimed at solving complex data challenges.

Medallion Architecture & Data Quality

Designed and implemented a scalable Lakehouse architecture using Databricks Unity Catalog. Validated, cleansed, and transformed data across Bronze, Silver, and Gold layers, ensuring 99.9% data quality through automated PySpark scripts.

DatabricksPySparkUnity CatalogAzure

Digitization & Automation Logic

Built a robust digitization solution leveraging Azure Data Factory and REST APIs. This automation reduced manual effort by 80% and improved process reliability for backend data workflows.

ADFREST APIsAutomationAzure

Real-time Streaming Pipeline

Developed high-velocity ingestion pipelines using Spark Structured Streaming and Kafka. Enabled real-time data processing and analytics for critical business insights.

KafkaSpark StreamingReal-timeBig Data