Senior Technical Lead
HCLTech · United States
Job Summary
Senior Data Engineer
Role purpose
We are looking for an experienced hands-on Senior Data Engineer to support the migration of existing PBWM PEGA-related data pipelines away from legacy SAS and Teradata processing and onto the AWS-based data platform. The engineer will pick up work already progressed earlier this year, complete the remaining SAS-to-PySpark migration, and implement the producer-side changes required to supply data into the agreed PEGA integration pattern.
Key Responsibilities
Key responsibilities
- Understand the existing SAS/Teradata pipelines supporting PBWM PEGA data provisioning and baseline the remaining migration scope.
- Convert remaining SAS transformation and business logic into production-quality PySpark.
- Develop and enhance batch data pipelines using Python/PySpark and AWS, particularly Glue and S3.
- Maintain equivalent business logic and data structures where appropriate to support a controlled migration.
- Reconcile PySpark outputs against the existing SAS/Teradata processes and support parallel-run/data validation.
- Work with PBWM, EDT Simplification, architecture and downstream PEGA engineering teams to resolve integration dependencies.
- Support testing, deployment, release and productionisation of the migrated pipelines.
- Document the solution and provide an effective handover to the permanent engineering team.
Skill Requirements
Essential skills
- Strong Python and PySpark
- AWS data engineering, particularly S3 and AWS Glue
- SQL and complex data transformation
- Experience migrating legacy ETL/data-processing workloads to cloud platforms
- Experience translating or reverse-engineering SAS workloads into Python/PySpark
- Batch data pipeline development
- Data reconciliation, testing and validation
- Git-based development and CI/CD
- Experience working in controlled enterprise production environments
Desirable
- SAS development experience
- Snowflake, Airflow/MWAA or EMR
- Teradata migration experience
- PEGA / Customer Decision Hub exposure
- Previous NatWest engineering experience would be advantageous
PEGA expertise isn't essential. The primary requirement is a strong Data Engineer who can take unfamiliar SAS code, understand the existing processing, reproduce it safely in PySpark, validate the results and take the solution through to production.