Must have

• Python

• PySpark

• Cloud

• AI

• Data engineering

• Data science

• Machine Learning

• Storage

• Spark

• Prototyping

• UNITY

• Databricks

• Hive

• Audit

• Communication skills

• English (B2)

Nice to have

• Degree

Requirements description

Your profile:

• Expert level knowledge of Python, with 2+ years of production PySpark experience;
• 1+ years of hands-on architecture experience on Databricks, with deep understanding of Unity Catalog and Hive megastore;
• 3+ years of technical leadership experience, driving engineering strategy and team delivery;
• Proven experience leading enterprise-scale data platform migrations, strong background in distributed system design handling petabyte-scale data;
• Solid understanding of data governance, compliance, and audit requirements;
• Experience designing robust APIs and integration patterns for enterprise environments;
• Passion for defensive programming and building scalable, reliable software;
• Excellent written and verbal communication skills, able to lead cross-functional collaboration;
• Strong problem-solving skills, strategic thinking, and decision-making under ambiguity;
• Good verbal and written communication skills in English (min B2);
• Availability tow ork with US East Coast time zone;

Nice-to-have:

• Experience as a Staff or Principal Engineer at a major technology company;
•  Contributions to open-source data platforms;
• Experience working directly with large enterprise customers in a sales engineering context;
• Degree in Computer Science, Engineering, or a related technical field (or equivalent practical experience)

Your responsibilities

1. Defining and evolving the technical roadmap for UCX migration workflows and platform architecture;
2. Driving key architectural decisions and ensuring consistency across storage, compute, and governance layers;
3. Leading the design and implementation of complex migration features and data workflows;
4. Optimizing Spark workloads and ensuring performance, reliability, and scalability of the platform;
5. Mentoring engineers, setting coding standards, and promoting best engineering practices;
6. Researching and prototyping new migration capabilities and Unity Catalog integrations;