Senior/Lead Data Software Engineer (Python, Spark, Azure)

Technologies we use

About the project

This is how we organize our work

This is how we work

Your responsibilities

• Migrate and optimize over 500 data jobs using Azure Databricks optimization techniques
• Manage and process 12 TB of data efficiently across platforms
• Tune machine learning models for Azure environments using Java Spark and Delta tables
• Update and maintain libraries to address security vulnerabilities
• Develop and maintain ETL/ELT pipelines using PySpark and related technologies
• Collaborate with cross-functional teams to integrate GenAI capabilities into data workflows
• Monitor data quality and implement improvements to ensure accuracy and reliability
• Automate deployment and operational tasks using Terraform and GitLab CI/CD
• Support data governance initiatives to comply with regulatory standards
• Troubleshoot and resolve performance issues in data processing systems
• Document system processes and provide technical guidance to junior engineers
• Implement best practices for code quality and data security
• Participate in code reviews and knowledge sharing sessions
• Optimize costs associated with data storage and processing

Our requirements

• Proficiency in Python and Spark with at least 3 years in data engineering roles
• Strong experience with Azure Databricks and PySpark
• Proven expertise in designing and implementing ETL/ELT solutions
• Experience migrating big data platforms to Azure-native services
• Proficiency with Delta tables for model tuning
• Knowledge of data governance and regulatory compliance frameworks
• Familiarity with Docker, Kubernetes (AKS), and Terraform for infrastructure automation
• Ability to manage large data volumes with high efficiency
• Excellent problem-solving and analytical skills
• Strong communication and collaboration abilities
• English proficiency at B2 level or higher

What we offer

This is how we work on a project

Development opportunities we offer