Requirements

Expected technologies

Python

Databricks

SQL

Microsoft Azure

Optional technologies

Airflow

Dagster

Apache Spark

Microsoft Power BI

Azure Event Hub

Kafka

Terraform

Operating system

Windows

macOS

Our requirements

• At least 5 years of commercial experience implementing, developing, or maintaining Big Data systems.
• Strong programming skills in Python: writing a clean code, OOP design.
• Experience in designing and implementing data governance and data management processes.
• Familiarity with Big Data technologies like Airflow or Dagster, Databricks, Spark, Azure Event Hub or Kafka.
• Experience implementing and deploying solutions in cloud environments (with a preference for Azure).
• Knowledge of how to build and deploy Power BI reports and dashboards for data visualization.
• Excellent understanding of dimensional data and data modeling techniques.
• Excellent communication skills and consulting experience with direct interaction with clients.
• Ability to work independently and take ownership of project deliverables.
• Master’s or Ph.D. in Computer Science, Data Science, Mathematics, Physics, or a related field.

Your responsibilities

• Design scalable data processing pipelines for streaming and batch processing using Big Data technologies like Databricks, Airflow and/or Dagster.
• Contribute to the development of CI/CD and MLOps processes.
• Develop applications to aggregate, process, and analyze data from diverse sources.
• Collaborate with the Data Science team on Machine Learning projects, including text/image analysis and predictive model building.
• Develop and organize data transformations using Databricks/DBT and Apache Airflow.
• Translate business requirements into technical solutions and ensure optimal performance and quality.