Stark.ai
About Me
- With over 4 years of experience in data engineering, specializing in ETL development, data migration and data architecture within the IT industry.
- Proficient in SQL, Pyspark, Python, Snowflake, cloud platforms like AWS, and data modeling, with a strong emphasis on implementing automated, cloud-based solutions.
- Familiar in designing and implementing scalable data pipelines using ETL/ELT frameworks to streamline data processing and analysis.
- Skilled in utilizing cloud platforms such as AWS, Azure, or GCP to build and manage data infrastructure for efficient storage, processing, and retrieval.
- Experienced in data modeling and schema design to optimize database structures for improved performance and usability.
- Effective communicator with the ability to collaborate cross-functionally, translating business needs into technical solutions and driving successful project outcomes.
Work Experience
Software Engineer
- EPAM
- Designing and optimizing data pipelines for a leading QSR (Quick Service Restaurant) enterprise, ensuring efficient data ingestion and processing in Snowflake.
- Enhancing expertise in PySpark transformations using Databricks to process large-scale restaurant operations and customer analytics data.
- Implementing AWS-based ETL solutions (Lambda, Step Functions, DMS) to streamline cloud data processing and improve operational efficiency.
- Resolving data inconsistencies between on-premises and cloud systems, ensuring data integrity across financial and operational reporting.
Dec 2024
-
Present
Pune
Senior Data Engineer
- Quantiphi Analytical Solutions Pvt. Ltd.
- Data Migration & Transformation: Data Migration & Transformation: Successfully led a migration from Teradata to Snowflake, achieving a 50% improvement in query performance and reducing data transformation times. Leveraged PySpark to efficiently transform and process large datasets during migration, ensuring data integrity and performance optimization. Developed and implemented streamlined, modular data transformations using dbt within Snowflake, enhancing data accessibility and boosting analytics performance. Converted legacy mappings into optimized Snowflake stored procedures for improved operational efficiency.
- Cloud-Based Integration: Architected and deployed ETL pipelines utilizing AWS Redshift, Lambda, Step Functions, and SNS to automate data ingestion into AWS Redshift, reducing manual effort by 40%.
- Reconciliation Framework: Engineered a reconciliation framework leveraging SQL, AWS Redshift and AWS Step Function, minimizing financial data discrepancies between on-premise and cloud environments.
- Optimized Data Processing: Designed complex SQL queries for data extraction and transformation, improving processing times by 30-40%.
- Python : Developed and optimized ETL pipelines using Python, and AWS services (Lambda, Redshift) to manage large-scale data processing across cloud environments.
Dec 2020
-
Nov 2024
Remote
Intern: Framework Engineer
- Quantiphi Analytical Solutions Pvt. Ltd.
- Orchestrated the development and maintenance of IICS pipelines, employing Python scripts to ensure accurate and efficient data ingestion into Redshift from multiple sources, with a focus on historical data load.
July 2020
-
Nov 2020
Remote
Education
BE - IT
2016
-
2020
Pimpri Chinchwad College of Engineering, Pune CGPA: 9.3
Projects
Restaurant Brands International
- Built data pipelines to ingest new tables into Snowflake.
- Designed test cases to validate data integrity and accuracy.
- Enhanced ETL using AWS Lambda, Step Functions, and DMS.
- Ensured on-premises and cloud data consistency, mitigating risks.
- Automated cloud infrastructure using Terraform.
Financial Data Transformation
- Engineered and deployed a reconciliation framework, reducing processing time by an impressive 50% compared to conventional on-premises methodologies.
- Enhanced the current ETL architecture by integrating AWS services such as Lambda, Step Function, DMS, Cloudwatch and SNS.
- Resolved data disparities between on-premises and cloud data warehouses, ensuring seamless data consistency and integrity, thereby mitigating potential financial risks.
Data Warehouse For Healthcare Application
- Devised and executed a consolidated architecture for Snowflake and Informatica Intelligent Cloud Services, refining historical and Change Data Capture (CDC) data ingestion.
- Elevated data integration procedures through the creation of a unified system.
- Engineered the user interface of a Streamlit application, enabling users to upload PDF files and engage with a large language model for real-time text processing and question answering, thereby enhancing user experience and accessibility.
Teradata to Snowflake Migration
- Executed a successful Teradata to Snowflake migration, achieving significant efficiency gains.
- Implemented Infoworks pipelines to manage large data volumes, reducing processing times from hours to 15-30 minutes for over 100 tables.
- Transformed data loading processes by converting Informatica BDM mappings into optimized Snowflake stored procedures, resulting in a 50% reduction in transformation time.
- Spearheaded the development and deployment of Autosys JIL scripts, enhancing system efficiency and ensuring data integrity through streamlined execution of Snowflake stored procedures.
Global Data Environment
- Orchestrated the development and maintenance of IICS pipelines, employing Python scripts to ensure accurate and efficient data ingestion into Redshift from multiple sources, with a focus on historical data load.
- Engineered intricate SQL queries for IICS mappings, optimizing data extraction from diverse source tables and reducing processing times by 30-40%, while enhancing the clarity of data transformation logic.
- Innovated the creation of sophisticated views for PowerBI dashboards, tailoring them to specific business requirements and enabling deeper insights and informed decision-making.
- Enhanced existing workflows to better align with evolving business needs, integrating advanced scripting and automation techniques for improved efficiency and adaptability.
Certifications
Snowflake Snowpro Core Certification
September 2021
-
AWS Associate Solution Architect
December 2020
-
