Lead Data Engineer
Accenture · Kochi
Job Title - Lead Data Engineer - Level 9 - ACS Song
Management Level: Level 9 – Lead/Specialist
Location: Kochi
Must have skills: Azure and Databricks
Good to have skills:
Experience: 5-8 years of experience is required
Educational Qualification: Graduation (Accurate educational details should capture)
Job Summary
We are seeking a Senior Data Engineer specializing in Azure and Databricks to design, build, and optimize scalable data engineering solutions across cloud-based and distributed data platforms. The role will focus on developing robust ETL/ELT pipelines, workflow orchestration, data processing frameworks, and backend automation using Python. The ideal candidate will bring strong hands-on experience with Databricks, Azure Functions, Apache Airflow, Kedro, Kafka, PostgreSQL, Docker, and modern software engineering practices (all must haves). This position requires close collaboration with architects, UI developers, and data engineering teams to deliver reliable, production-ready, and high-performing data solutions.
Roles and Responsibilities
- Design and deliver scalable, maintainable, and production-grade data pipelines and data processing workflows using Azure, Databricks, Python, and related technologies.
- Enable efficient data ingestion, transformation, orchestration, validation, and automation across batch and streaming data environments.
- Support the development of modular, reusable, and well-documented data engineering components using frameworks such as Kedro and Apache Airflow.
- Ensure high standards of code quality, performance, reliability, troubleshooting, and collaboration within cross-functional engineering teams.
- Develop, maintain, and enhance scalable data pipelines, ETL/ELT workflows, and data processing solutions on Azure and Databricks.
- Work extensively with Kedro for pipeline development, modularization, feature implementation, code changes, and bug fixes.
- Build, maintain, and optimize workflow orchestration using Apache Airflow, including DAG creation, monitoring, troubleshooting, and issue resolution.
- Develop and maintain Azure Functions using Python for backend processing, integrations, automation, and event-driven workflows.
- Write clean, scalable, maintainable, and production-ready Python code, including use of Pydantic models, data validation, modular design, and API/backend integrations.
- Design, optimize, and troubleshoot complex SQL queries and transformations, with hands-on work in PostgreSQL for database development and data management.
- Work with Kafka and event-driven or streaming architectures for data ingestion, processing, and integration workflows.
- Use Docker for containerization and support local development, testing, and deployment workflows.
- Use GitHub for version control, pull requests, code reviews, branching strategies, and collaborative software development.
- Collaborate with architects, UI developers, data engineers, and other stakeholders while participating in sprint planning, technical discussions, code reviews, documentation, testing, CI/CD, monitoring, and production support activities
Professional and Technical Skills:
- Minimum 6 years of total professional experience in data engineering, cloud data platforms, backend data processing, or related software engineering roles.
- Strong experience designing, developing, and supporting production-grade data pipelines and ETL/ELT workflows.
- Hands-on experience working in enterprise or production environments with cloud-based data engineering solutions.
- Proven ability to debug, troubleshoot, and resolve application issues, pipeline failures, DAG issues, and production bugs.
- Strong programming experience in Python for data engineering, backend processing, pipeline development, and automation.
- Good understanding of Python programming concepts, including modular code development, Pydantic models, data validation, backend integrations, and API-based workflows.
- Hands-on production experience with Kedro for pipeline development, modularization, and maintainable data engineering design.
- Strong hands-on experience with Apache Airflow, including DAG creation, workflow orchestration, scheduling, monitoring, and troubleshooting.
- Experience with Databricks and distributed data processing environments.
- Hands-on experience developing and maintaining Azure Functions using Python.
- Strong working knowledge of the Microsoft Azure ecosystem and cloud-based data engineering services.
- Experience with Kafka and event-driven or data streaming architectures.
- Strong SQL skills with hands-on experience in PostgreSQL, including query optimization, data validation, and database development.
- Hands-on experience with Docker and containerized development or deployment workflows.
- Experience using GitHub for collaborative development, pull requests, code reviews, and version control.
Additional Information
Behavioral and Collaboration Skills
- Strong analytical, troubleshooting, and problem-solving skills.
- Ability to work effectively in cross-functional engineering teams.
- Strong communication skills (English) with the ability to participate in technical discussions and explain implementation approaches clearly.
- Comfortable collaborating with architects, UI developers, data engineers, and other stakeholders.
- Proactive mindset with ownership of assigned tasks, production issues, and continuous improvement initiatives.
About Our Company | Accenture (do not remove the hyperlink)
- Minimum 6 years of total professional experience in data engineering, cloud data platforms, backend data processing, or related software engineering roles.
- Strong experience designing, developing, and supporting production-grade data pipelines and ETL/ELT workflows.
- Hands-on experience working in enterprise or production environments with cloud-based data engineering solutions.
- Proven ability to debug, troubleshoot, and resolve application issues, pipeline failures, DAG issues, and production bugs.
- Strong programming experience in Python for data engineering, backend processing, pipeline development, and automation.
- Good understanding of Python programming concepts, including modular code development, Pydantic models, data validation, backend integrations, and API-based workflows.
- Hands-on production experience with Kedro for pipeline development, modularization, and maintainable data engineering design.
- Strong hands-on experience with Apache Airflow, including DAG creation, workflow orchestration, scheduling, monitoring, and troubleshooting.
- Experience with Databricks and distributed data processing environments.
- Hands-on experience developing and maintaining Azure Functions using Python.
- Strong working knowledge of the Microsoft Azure ecosystem and cloud-based data engineering services.
- Experience with Kafka and event-driven or data streaming architectures.
- Strong SQL skills with hands-on experience in PostgreSQL, including query optimization, data validation, and database development.
- Hands-on experience with Docker and containerized development or deployment workflows.
- Experience using GitHub for collaborative development, pull requests, code reviews, and version control.
About Accenture
Accenture is a leading global professional services company that helps the world’s leading businesses, governments and other organizations build their digital core, optimize their operations, accelerate revenue growth and enhance citizen services—creating tangible value at speed and scale. We are a talent- and innovation-led company with approximately 791,000 people serving clients in more than 120 countries. Technology is at the core of change today, and we are one of the world’s leaders in helping drive that change, with strong ecosystem relationships. We combine our strength in technology and leadership in cloud, data and AI with unmatched industry experience, functional expertise and global delivery capability. Our broad range of services, solutions and assets across Strategy & Consulting, Technology, Operations, Industry X and Song, together with our culture of shared success and commitment to creating 360° value, enable us to help our clients reinvent and build trusted, lasting relationships. We measure our success by the 360° value we create for our clients, each other, our shareholders, partners and communities.Visit us at www.accenture.com
Equal Employment Opportunity Statement
We believe that no one should be discriminated against because of their differences. All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, military veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by applicable law. Our rich diversity makes us more innovative, more competitive, and more creative, which helps us better serve our clients and our communities.