Principal Data Engineer at Bengaluru (Bangalore) Full Time

Key Responsibilities

• Design, develop, and optimize data pipelines using Apache Spark to process large volumes of structured and unstructured data.
• Write efficient and maintainable code in Scala and Python for data extraction, transformation, and loading (ETL) operations.
• Collaborate with cross-functional teams to define data engineering solutions to support analytics and machine learning initiatives.
• Implement and maintain data lake and warehouse solutions using cloud platforms (e.g., AWS, GCP, Azure).
• Ensure data workflows and distributed systems’ performance, scalability, and reliability.
• Perform data quality assessments, implement monitoring, and improve data governance practices.
• Assist in migrating and refactoring legacy data systems into modern distributed data processing platforms.
• Provide technical leadership and mentorship to junior engineers and contribute to best practices in coding, testing, and deployment.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 8+ years of hands-on experience in data engineering, with strong skills in Apache Spark, Scala, and Python.
• Experience with distributed data processing frameworks and real-time data processing.
• Strong experience with big data technologies such as Hadoop, Hive, and Kafka.
• Proficient with relational databases (SQL, PostgreSQL, MySQL) and NoSQL databases (Cassandra, HBase, MongoDB). 
• Knowledge of CI/CD pipelines and DevOps practices for deploying data workflows.
• Strong problem-solving skills and experience with optimizing large-scale data systems.
• Excellent communication and collaboration skills.
• Experience with orchestration tools like Airflow
• Experience with containerization and orchestration (e.g., Docker, Kubernetes)