Data Engineer – Lightning job by Cutshort ⚡ at Mumbai Full Time

Lightning job by Cutshort ⚡:

As part of this feature, you can expect status updates about your application and replies within 48 hours (once the screening questions are answered)

About us:

Accrete is a fast-growing enterprise software company that’s at a major inflection point after completing our Series A raise. Just four years ago, our founders combined their experience, over a decade of proprietary artificial intelligence technology and patents to start Accrete, and we were off to the races. Now our team is 100+ members strong and we have delivered major solutions to name-brand companies in music/entertainment, finance, collectibles, and government.

About the role:

As a Data Engineer specializing in AWS Cloud and Airflow, you will be responsible for designing, building, and maintaining data pipelines and infrastructure on the AWS cloud platform using Apache Airflow. Your expertise in AWS services and Airflow will enable the efficient processing, transformation, and storage of large volumes of data to support data-driven decision-making across the organization.

Responsibilities:

• Data Pipeline Development: Design, implement, and manage end-to-end data pipelines using Apache Airflow to extract, transform, and load data from various sources into AWS data stores (S3, Redshift, etc.).
• AWS Cloud Services: Utilize a wide range of AWS services, including but not limited to EC2, S3, Lambda, Glue, EMR, and Redshift, to build scalable and cost-effective data solutions.
• Data Transformation and Processing: Develop ETL workflows using Airflow to cleanse, enrich, and transform raw data into formats suitable for analysis and reporting.
• Data Storage and Warehousing: Design and maintain data storage solutions on AWS, ensuring data is stored securely and efficiently.
• Monitoring and Optimization: Monitor data pipelines and AWS infrastructure performance, identify bottlenecks, and optimize data processing for improved efficiency and reliability.
• Data Quality Assurance: Implement data quality checks and validation processes to ensure data accuracy and integrity throughout the data engineering pipeline.
• Data Security and Compliance: Implement security measures and best practices to protect sensitive data and comply with data privacy regulations.
• Collaboration and Documentation: Collaborate with cross-functional teams, including data scientists and analysts, to understand data requirements and deliver data solutions. Maintain documentation for data engineering processes and technical specifications.
• Automation and Orchestration: Automate repetitive tasks and data workflows using Airflow DAGs, reducing manual intervention and improving workflow efficiency.
• Troubleshooting and Support: Identify and resolve data pipeline issues, provide technical support, and assist in root cause analysis.
• Continuous Learning: Stay up-to-date with the latest AWS services, Airflow updates, and data engineering best practices to drive innovation and implement improvements.

Skills and Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer with hands-on experience in AWS cloud services and Apache Airflow.
• Strong proficiency in Python for scripting and data manipulation tasks.
• In-depth knowledge of AWS services for data processing, storage, and analytics.
• Familiarity with big data technologies such as Hadoop, Spark, and NoSQL databases is a plus.
• Experience with data warehousing concepts and data modeling.
• Strong understanding of ETL principles and data integration techniques.
• Knowledge of version control systems like Git for code management and collaboration.
• Excellent problem-solving and analytical skills with the ability to handle complex data engineering challenges.
• Effective communication and teamwork skills to work collaboratively with various stakeholders.