Senior Data Engineer

CF-IT's TOP organization is seeking a Senior Data Engineer, located at the Hyderabad IT Hub. Specifically, you'll work within the Data & AI section that supports the Modern IT Service & Asset Management team.

Responsibilities:

Leading design and development of data and analytics cloud-based platform. Crafting integrated systems, implementing ELT/ ETL jobs to fulfil business deliverables. Performing sophisticated data operations such as data orchestration, transformation, and visualization with large datasets. You will be working with product managers to ensure superior product delivery to drive business value & transformation. Demonstrating standard coding practices to ensure delivery excellence and reusability.

• Data Ingestion: Develop and maintain data pipelines to extract data from various sources and load it into Azure and Databricks environments.

• Data Transformation: Design and implement data transformation processes, including data cleansing, normalization, and aggregation, to ensure data quality and consistency.

• Data Modeling: Develop and maintain data models and schemas to support efficient data storage and retrieval in Azure and Databricks platforms.

• Data Warehousing: Design and build data warehouses or data lakes using Azure services such as Azure Data Lake Storage, Databricks DeltaLake

• Data Integration: Integrate data from multiple sources, both on-premises and cloud-based, using Azure Data Factory or other relevant tools.

• Data Governance: Implement data governance practices, including data security, privacy, and compliance, to ensure data integrity and regulatory compliance.

• Performance Optimization: Optimize data pipelines and queries for improved performance and scalability in Azure and Databricks environments.

• Monitoring and Troubleshooting: Monitor data pipelines, identify and resolve performance issues, and troubleshoot data-related problems in collaboration with other teams.

• Data Visualization: Build BI reports to enable faster decision making.

• Collaboration: Work with product managers to ensure superior product delivery to drive business value & transformation

• Documentation: Document data engineering processes, data flows, and system configurations for future reference and knowledge sharing.