Position overview

We are looking for a Data Engineer to help build the next generation of our cloud-based data platform using AWS and Databricks. In this role, you will design and operate scalable, resilient, high-quality data pipelines and services that empower analytics, real-time streaming, and machine learning use cases across the organization.

Responsibilities

• Design, build, and operate robust, scalable, secure data pipelines across batch, streaming, and real-time workloads.
• Transform raw data into high-quality, reusable datasets and data products that power analytics and ML.
• Work hands-on with AWS, Apache Flink, Databricks, PySpark/Spark SQL, and modern data tooling.
• Develop ETL/ELT processes, ingestion patterns, and streaming integrations using services such as Kafka, Kinesis, Glue, Lambda, EMR, DynamoDB, and Athena.
• Ensure data reliability and observability through monitoring, alerting, testing, and CI/CD best practices.
• Drive engineering best practices in performance tuning, cost optimisation, security, metadata management, and data quality.
• Partner with Data Product Owners, ML teams, and business stakeholders to translate requirements into technical solutions.
• Lead technical design discussions, influence data platform decisions, and mentor other engineers.
• Operate services in production with a focus on uptime, data availability, and continuous improvement.

Requirements

• 4+ years of experience building data pipelines and large-scale ETL/ELT workflows.
• Strong hands-on experience with AWS cloud data services, Apache Flink, and Databricks ecosystem.
• Deep proficiency in Java, Python, PySpark/Spark SQL, SQL optimisation and performance tuning.
• Experience with streaming architectures: Kafka, Kinesis, or similar.
• Familiarity with CI/CD, infrastructure-as-code, automation, and DevOps practices.
• Experience with data warehousing, structured and semi-structured data, and performance-optimised storage formats (Parquet/Delta).
• Knowledge of Agile development and modern engineering practices.

Nice to have

• Experience with Machine Learning data pipelines, feature stores, or MLOps.
• Familiarity with data governance, data cataloging, lineage, and metadata tools.
• Experience with containerization and orchestration (Docker, ECS, Kubernetes, Airflow, Step Functions).
• Knowledge of scalable data warehousing technologies.
• Contributions to engineering communities, open-source, or internal tech groups.