Senior Lead Data Engineer - Databricks
Integrant · Maadi
Description
We're on the lookout for exceptional individuals to join our team as Senior Lead Data Engineers – Databricks. As part of our team, you'll play a key role in designing and delivering modern, scalable data platforms for clients across multiple industries, leveraging Azure and Databricks technologies.
You will lead the design, development, and optimization of end-to-end data engineering solutions built on modern Lakehouse architectures. Working closely with clients and cross-functional teams, you'll translate business requirements into scalable, high-performance data platforms while mentoring engineers and driving technical excellence.
Requirements
- Responsibilities
- Lead the design, development, and maintenance of scalable data engineering solutions using Azure and Databricks.
- Design and implement modern Lakehouse architectures following industry best practices.
- Build and optimize large-scale batch and streaming data pipelines.
- Develop ETL/ELT pipelines using Databricks, PySpark, SQL, and Azure Data Factory.
- Design, implement, and optimize Delta Lake solutions for reliable and performant data processing.
- Collaborate with clients to gather requirements and translate business needs into technical solutions.
- Optimize Spark workloads for performance, scalability, and cost efficiency.
- Design and maintain enterprise-grade Data Warehouses, Data Lakes, and Lakehouses.
- Implement data governance, security, and metadata management using Azure services such as Purview, Key Vault, and Microsoft Entra ID (Azure Active Directory).
- Establish CI/CD pipelines and DevOps practices for data engineering workloads.
- Mentor and coach junior data engineers while promoting engineering best practices.
- Participate in architecture discussions, solution design, and technical leadership initiatives.
- Contribute to internal knowledge sharing and continuous improvement.
- Requirements
- Required
- 10+ years of professional experience in Data Engineering.
- 4+ years of hands-on experience with Databricks.
- Strong experience developing data pipelines using PySpark and Apache Spark.
- Strong experience designing and implementing Lakehouse architectures.
- Experience working with Delta Lake and Medallion Architecture.
- Strong knowledge of SQL and Python.
- Experience with Azure Data Factory.
- Experience working with Azure Data Lake Storage Gen2 (ADLS Gen2).
- Experience with Azure Synapse Analytics.
- Experience with Data Warehousing concepts and dimensional modeling.
- Strong understanding of ETL and ELT design patterns.
- Experience optimizing Spark jobs, partitioning strategies, joins, caching, and workload performance.
- Experience with Git and Azure DevOps CI/CD pipelines.
- Experience implementing secure data platforms using Azure Key Vault, Microsoft Entra ID, and Microsoft Purview.
- Experience building scalable enterprise data platforms in Azure.
- Experience working directly with clients and gathering technical requirements.
- Experience mentoring engineers and leading technical initiatives.
- Strongly Preferred
- Experience with Snowflake.
- Experience building real-time streaming solutions using Structured Streaming, Kafka, or Event Hubs.
- Experience with Databricks Workflows.
- Experience with Delta Live Tables.
- Experience with Unity Catalog.
- Experience with Databricks SQL.
- Experience with MLflow.
- Experience with dbt.
- Experience with Airflow or other orchestration platforms.
- Experience working in Agile environments.
- Nice to Have
- MSc in Computer Science or a related field.
- Experience with DataOps practices.
- Experience with MLOps.
- Experience with Azure Machine Learning.
- Experience with Azure Cognitive Services.
- Experience with Docker and Kubernetes.
- Experience working with Microsoft Fabric.
- Experience with Power BI.
- Technical Skills
- Core Technologies
- Azure Databricks
- Apache Spark
- PySpark
- SQL
- Python
- Delta Lake
- Azure Data Factory
- Azure Synapse Analytics
- ADLS Gen2
- Snowflake
- Data Engineering
- Data Warehousing
- Data Lakes
- Lakehouse Architecture
- Medallion Architecture
- Dimensional Modeling
- ETL / ELT
- Batch Processing
- Streaming Data Pipelines
- Data Governance
- Performance Optimization
- Azure Ecosystem
- Azure DevOps
- Microsoft Purview
- Azure Key Vault
- Microsoft Entra ID
- Azure Functions
- Event Hubs
- Nice-to-Have Technologies
- Kafka
- Airflow
- dbt
- MLflow
- Docker
- Kubernetes
- Power BI
Benefits
- Salary paid in USD
- Six-month career advancing opportunities
- Supportive and friendly work environment
- Premium medical insurance [employee +family]
- English language development courses
- Interest-free loans paid over 2.5 years
- Technical development courses
- Planned overtime program (POP)
- Employment referral program
- Premium location in Maadi
- Social insurance