About the job
A full stack data engineer to build highly scalable, high quality data products, data quality routines and identifies and manages reference data. Creation of frameworks for spoke teams for ease data ingestion and processing from on-prem to cloud. Creation and maintainance of optimum data pipeline architecture for ingestion, storage, processing & transformation of data for building data products for analytics. Ensure Audit, Balance & Control are implemented for Consistent, Complete & Comprehensive Data Quality and Data Integrity management. Ensure compliance to data strategy and governance and integrate organization’s data management processes into data product development. Accountable for implementing modern-day security controls for sensitive data products. Write automated unit test cases for data products and integrate to CI/CD pipeline. Creation of CI/CD pipelines in Azure DevOps. Build solutions which are scalable, resilient and sustainable to address business requirements. Data engineering with big data using technologies such as Spark, Python, Azure Data Factory, Azure Data Lake Store, Azure Databricks, SQL, Unix, Postgresql, Hive, Azure DevOps. Data modelling, data pipeline design & development. Performance tuning and query optimization on large data sets. Knowledge about data virtualization, semantic layer tool like Dremio, Data flow diagrams, Data dictionary, Database normalization theory techniques, Entity relation modelling, Datawarehouse concepts and design techniques.