Senior Data Engineer – Scala / Spark / Hadoop (f/m)

We are looking for an experienced Data Engineer to join an international team working on a strategic regulatory initiative within the financial sector. You will develop and maintain a data platform and reporting processes operating in a complex Big Data environment.

Responsibilities:

Design, develop and maintain Data Engineering solutions using Hadoop and Apache Spark.

Build scalable solutions for processing large volumes of data.

Analyse complex business requirements and translate them into effective technical solutions.

Develop and optimise SQL queries and data-processing workflows.

Build and enhance data pipelines and reporting processes.

Create and maintain workflows using Apache Oozie and Apache Airflow.

Develop data solutions using Scala.

Work with event-driven architecture based on Apache Kafka.

Maintain and improve CI/CD processes using Git, pull requests and Jenkins.

Provide L3 support for production environments.

Investigate production incidents and technical problems in cooperation with relevant operational teams.

Collaborate with business stakeholders to define requirements and analyse issues.

Ensure the quality, performance, security and reliability of data solutions.

Requirements:

Strong hands-on experience with Scala.

Commercial experience with Apache Hadoop.

Practical knowledge of Apache Spark and large-scale data processing.

Experience building and maintaining Data Engineering pipelines.

Experience with Apache Kafka and event-driven architecture.

Practical knowledge of CI/CD processes, particularly Jenkins.

Very good knowledge of Git, code review and pull request processes.

Strong SQL development and optimisation skills.

Experience supporting production environments.

Ability to independently investigate complex technical issues.

English proficiency sufficient for daily work in an international team.

Availability to work from the Warsaw or Gdańsk office twice per week.

Ability to perform the work from Poland.

Nice to have:

Kubernetes experience.

Knowledge of Apache Flink.

Experience with Apache Airflow or Apache Oozie.

Previous involvement in regulatory or reporting projects.

Experience in the financial sector.

What we offer:

Participation in a strategic regulatory initiative with a tangible impact on the financial sector.

Work within a large and advanced Big Data environment.

A modern technology stack including Scala, Hadoop, Spark, Kafka, Airflow and Jenkins.

A high level of independence and influence over technical solutions.

Cooperation with an experienced international team.

Stable, long-term cooperation.

A professional environment focused on quality, security and expert development.

An opportunity to gain experience in financial regulations and regulatory reporting.