Azure Data Engineer
Aon · Poland
Azure Data Engineer
As an Azure Databricks Data Engineer, you will work with the team to design and deliver scalable data pipelines, Lakehouse solutions, and governed data platforms on Azure. You will apply your expertise in PySpark, SQL, Delta Lake, Unity Catalog, and Power BI enablement to create secure, reliable, and high-performing data solutions.
Our office is located in Krakow and this role is available both in 50/50 hybrid and fully remote working model.
Aon is in the business of better decisions
At Aon, we shape decisions for the better to protect and enrich the lives of people around the world. As an organization, we are united through trust as one inclusive team and we are passionate about helping our colleagues and clients succeed.
What the day will look like
Your responsibilities will include:
- Facilitate workshops to discuss, review and document business reporting requirements.
- Establish best practices across data architecture, governance, security, performance, and platform reliability
- Developing and implementing scalable data integration pipelines on Azure Databricks using PySpark, SQL, Notebooks, Jobs, and Workflows.
- Building and maintaining Lakehouse architectures using Delta Lake and bronze, silver, and gold data layers to support analytics and reporting.
- Crafting robust, reusable systems and collections for data transformation.
- Implementing and managing Unity Catalog to provide centralized governance of data, tables, and permissions.
- Defining and enforcing role-based access control, data access controls, and auditing in line with security and compliance requirements.
- Standardizing catalog, schema, and table naming conventions across environments.
- Delivering well-modelled, performance-optimized datasets for Power BI and other business analytics tools.
- Collaborating with BI developers and analysts to define semantic models, data contracts, and refresh strategies. Supporting design and development of new reports
- Troubleshooting and optimizing query performance between Power BI and Databricks, including Direct Lake and Direct Query where applicable.
- Configuring and improving Databricks clusters, cluster policies, and job schedules for cost, performance, and reliability.
- Implementing monitoring, logging, and alerting for data pipelines and jobs.
- Contributing to and improving CI/CD pipelines for Databricks code using Git and Azure DevOps or GitHub.
- Explore AI/ML technologies and opportunities to deploy machine learning pipelines that generate measurable business value.
How this opportunity is different
This role provides the opportunity to develop scalable and governed data solutions that directly support enterprise analytics and business decision-making. You will combine hands-on engineering with technical leadership, helping to establish best practices across data architecture, governance, security, performance, and platform reliability.
Skills and experience that will lead to success
- Knowledge of Finance and Insurance business processes and terminology.
- Work independently, Self-motivated and organized, Manages own time to ensure task are completed with the ability to work to tight deadlines and under high pressure situations
- Can communicate openly, honestly and independently with stakeholders at all levels of the organisation to understand their business goals and processes.
- Strong experience crafting and implementing scalable ETL and ELT pipelines on Azure Databricks using PySpark, SQL, Notebooks, Jobs, and Workflows.
- Expertise in Lakehouse architecture, Delta Lake, and bronze, silver, and gold data-layer patterns, with the ability to develop reusable data transformation frameworks and libraries.
- Experience implementing Unity Catalog, role-based access control, data access controls, auditing, and consistent catalog, schema, and table standards.
- Ability to develop performance-optimized datasets for Power BI and collaborate on semantic models, data contracts, refresh strategies, Direct Lake, and Direct Query solutions where applicable.
- Experience optimizing Databricks clusters and job scheduling, implementing monitoring and alerting, and contributing to CI/CD practices using Git and Azure DevOps or GitHub.
Please attach CV in English only.
#technologyandsecurity
#li-mk1
#LI-remote