Data Engineer (Azure)

Technologies we use

About the project

Your responsibilities

• Build reusable, metadata-driven data pipelines.
• Automate and optimize data platform processes.
• Develop integrations with data sources and consumers.
• Extend shared ETL libraries with transformation methods.
• Write unit tests.
• Create monitoring solutions for the Databricks platform.
• Proactively address ETL performance and quality issues.
• Collaborate with infrastructure teams on cloud resources.
• Update data platform wiki and documentation.
• Conduct code reviews to ensure quality.
• Initiate and implement architecture improvements.

Our requirements

• Proficiency in Python/PySpark and SQL.
• Strong experience building robust data pipelines with Databricks Spark.
• Proven track record handling large, complex datasets.
• Expertise in developing reusable data transformation libraries (Python packages).
• Deep knowledge of Databricks Delta optimization (partitioning, Z-ordering, compaction, etc.).
• Hands-on experience with CI/CD pipeline development.
• Skilled in event streaming integration using Kafka, Event Hubs, or Kinesis.
• Solid understanding of fundamental networking concepts.
• Familiarity with Agile/Scrum methodologies.

Optional

• Spark Structured Streaming, IaC (Terraform/Bicep), containers (Azure Container Apps/K8s), event sourcing, CDC (Debezium), Azure native (Data Factory, Functions, ACI)

This is how we organize our work

This is how we work

Team members

This is how we work on a project

What we offer

• 100 Remote work model.
• Superior co-working and personal development experience in an international setting.