Must have

• Python

• PySpark

• Databricks

• UNITY

• Hive

• English (B2)

• Polish (B1)

Requirements description

• Expert-level knowledge of Python, with 2+ years of production PySpark experience,

• 1+ years of hands-on architecture experience on Databricks, with deep understanding of Unity Catalog and Hive metastore,

• 3+ years of technical leadership experience, driving engineering strategy and team delivery,

• proven experience leading enterprise-scale data platform migrations,

• strong background in distributed system design handling petabyte-scale data,

• solid understanding of data governance, compliance, and audit requirements,

• experience designing robust APIs and integration patterns for enterprise environments,

• passion for defensive programming and building scalable, reliablesoftware,

• excellent written and verbal communication skills, able to lead cross-functional collaboration,

• strong problem-solving skills, strategic thinking, and decision-making under ambiguity,

• good verbal and written communication skills in English (min. B2),

• availability to work with US East Coast time zone.

Offer description

Fully remote work.

Client from: USA

New project.

DESCRIPTION

Client is a cloud-based data and AI platform that helps companies store, process, and analyze large amounts of data. It combines data engineering, data science, and machine learning in one unified workspace. Teams can easily collaborate using notebooks and shared workflows. Many organizations use their platform to build scalable analytics and AI solutions in the cloud.

Your responsibilities

1. Defining and evolving the technical roadmap for UCX migration workflows and platform architecture,
2. driving key architectural decisions and ensuring consistency across storage, compute, and governance layers,
3. leading the design and implementation of complex migration features and data workflows,
4. optimizing Spark workloads and ensuring performance, reliability and scalability of the platform,
5. mentoring engineers, setting coding standards, and promoting best engineering practices,
6. researching and prototyping new migration capabilities and Unity Catalog integrations.