Sr Data Engineer - Spark-Scala - Chennai

Photon · India

Apply on the company website

  • Develop, test, and deploy data processing applications using Apache Spark and Scala. 
  • Optimize and tune Spark applications for better performance on large-scale data sets. 
  • Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Kafka) to build data pipelines and storage solutions. 
  • Collaborate with data scientists, business analysts, and other developers to understand data requirements and deliver solutions. 
  • Design and implement high-performance data processing and analytics solutions. 
  • Ensure data integrity, accuracy, and security across all processing tasks. 
  • Troubleshoot and resolve performance issues in Spark, Cloudera, and related technologies. 
  • Implement version control and CI/CD pipelines for Spark applications. 

Required Skills & Experience: 

  • Minimum 8 years of experience in application development. 
  • Strong hands on experience in Apache Spark, Scala, and Spark SQL for distributed data processing. 
  • Hands-on experience with Cloudera Hadoop (CDH) components such as HDFS, Hive, Impala, HBase, Kafka, and Sqoop. 
  • Familiarity with other Big Data technologies, including Apache Kafka, Flume, Oozie, and Nifi. 
  • Experience building and optimizing ETL pipelines using Spark and working with structured and unstructured data. 
  • Experience with SQL and NoSQL databases such as HBase, Hive, and PostgreSQL. 
  • Knowledge of data warehousing concepts, dimensional modeling, and data lakes. 
  • Ability to troubleshoot and optimize Spark and Cloudera platform performance. 

  • Familiarity with version control tools like Git and CI/CD tools (e.g., Jenkins, GitLab). 

    Compensation, Benefits and Duration

    • Minimum Compensation: USD
    • Maximum Compensation: USD
    • Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.
    • Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.
    • This position is not available for independent contractors
    • No applications will be considered if received more than 120 days after the date of this post

More jobs at Photon