Senior Technical Lead
HCLTech · Dubai
Job Summary
A PySpark Developer is responsible for designing, developing, and maintaining large-scale, distributed data processing systems and pipelines. They leverage Apache Spark’s Python API (PySpark) to ingest, transform, and analyze massive volumes of structured and unstructured data.
Core Responsibilities
- Data Pipeline Development: Design and build scalable, efficient ETL/ELT pipelines to move and transform data from various sources into data lakes or warehouses.
- Big Data Processing: Write optimized PySpark scripts (using DataFrames, Datasets, and Spark SQL) to process terabytes/petabytes of data.
- Performance Tuning: Troubleshoot, debug, and optimize Spark jobs by managing partitions, caching, broadcast variables, and minimizing data shuffling.
- Data Modeling: Assist in designing data architectures, schemas, and storage formats (e.g., Parquet, Delta Lake, ORC).
- Integration: Connect Spark applications with cloud storage (AWS S3, Azure ADLS, Google Cloud Storage) and databases (NoSQL, relational).
- CI/CD & Automation: Deploy data workflows using orchestration tools and maintain code quality through unit testing.
Technical Skills Required
| Category | Skills & Technologies |
|---|---|
| Core Languages | Python (Advanced), SQL (Advanced) |
| Big Data Frameworks | Apache Spark / PySpark, Hadoop, Hive |
| Cloud Platforms | AWS (EMR, Glue), Azure (Databricks, Synapse), or GCP (Dataproc) |
| Storage & Formats | Delta Lake, Parquet, Avro, HDFS, Snowflake, Amazon S3 |
| Orchestration | Apache Airflow, Prefect, or Luigi |
Key Responsibilities
as above
Skill Requirements
as above