Data Engineer - Site Reliability

Digital Hub Warsaw at Bayer

Warszawa, Ochota
21000 zł/mth.
Hybrid
🐍 Python
Golang
PostgreSQL
PostGIS
🔍 Google Cloud Vertex AI
☁️ AWS SageMaker
☁️ AWS
🚢 Kubernetes
Terraform
GCDM
CloudFormation
🐳 Docker compose
SQL
Hybrid

Requirements

Expected technologies

Python

Golang

PostgreSQL

PostGIS

Google Cloud Vertex AI

AWS SageMaker

AWS

Kubernetes

Terraform

GCDM

CloudFormation

Docker compose

SQL

Our requirements

  • Bachelor's degree in Computer Science, Engineering, or relevant job experience with 5+ years
  • Strong proficiency in Infrastructure as Code (IaC) principles and tools (i.e. Terraform, GCDM, CloudFormation, Docker compose, etc.)
  • Strong understanding and hands-on experience with CI/CD tools such as GitHub Actions, Terraform, Google Cloud Build, ArgoCD
  • Strong proficiency in Python and/or Go programming languages
  • Hands-on experience working with Kubernetes(K8s) for orchestrating and managing containerized data services and workflows.
  • Proven experience with SQL databases and/or BigQuery, including schema design and query optimization
  • Hands-on experience with event-driven and streaming data architectures using platform and services such as Kafka, AWS SQS, and Google Cloud Pub/Sub
  • Ability to work in a highly collaborative environment and to communicate effectively with internal and external partners
  • Experience working with and testing high-volume RESTful APIs deployed on common cloud Kubernetes infrastructures
  • Experience with utilizing Docker to build and deploy containerized applications
  • Experience with telemetry data flows and monitoring and logging tools such as Grafana, Prometheus, ELK stack, or equivalent.
  • Experience in cloud platforms such as GCP and AWS, including native data and compute services such as Bigquery/Aurora, GCS/S3, GKE/EKS, GCE/EC2, Cloud Functions/Lambda
  • Experience with code versioning and dependency management systems such as GitHub
  • Excellent problem-solving skills and the ability to work effectively in a fast-paced, collaborative environment.
  • Strong communication skills and the ability to articulate technical concepts to non-technical stakeholders.
  • Preferred:
  • Highly proficient (5+ years) in Python or Golang with a strong track record of maintaining production data pipelines and backend systems
  • Experience with PostgreSQL/PostGIS enabled databases
  • Experience scaling GRPC services, working with protobuf schemas and tooling such as BSR
  • Experience with object-oriented design, coding and testing patterns, and implementing complex data projects in a large-scale data infrastructure
  • Understanding of geospatial data concepts. Experience with data processing and analysis using geospatial libraries and tools
  • Familiarity with cloud-based machine learning services and platforms such as Google Cloud Vertex AI or AWS SageMaker. Experience with deploying invoking model endpoints.
  • Solid understanding of networking concepts, security principles, and best practices for cloud environments
  • Experience working with customers and developers to deliver full-stack development solutions; the ability to translate customer requirements into technical requirements in an Agile environment
  • You feel you do not meet all criteria we are looking for? That doesn’t mean you aren’t the right fit for the role. Apply with confidence, we value potential over perfection!

Your responsibilities

  • Drive continual improvement in deployment, observability, monitoring, and scalability
  • Automate tasks to improve efficiency, manage infrastructure, and ensure performance of Loc360 assets that meet or exceed stated SLOs
  • Assist developers implementing scalable data pipelines in Python and Go for ingestion, transformation, and delivery of structured and unstructured geospatial data.
  • Write and review code, develop documentation, and debug complex problems between systems and components.
  • Optimize event-driven data processing solutions using Kafka, AWS SQS, and Google Cloud Pub/Sub to orchestrate multi-stage spatial workflows.
  • Integrate and manage data flows across cloud platforms such as AWS and GCP, databases such as PostgreSQL/PostGIS and BigQuery, and cloud storage such as AWS S3 and Google Cloud Storage
  • Leverage Kubernetes (K8s) for deploying and managing containerized applications and workflows.
  • Work closely with cloud engineers and platform engineers to integrate cloud infrastructure with CI/CD pipelines and standardize deployment patterns.
  • Collaborate with data engineers to optimize and monitor data pipelines and services for performance, reliability, scalability, and cost-effectiveness
  • Provide technical support, including incident response, troubleshooting and resolution for production issues in data pipelines and API services
  • Ensure compliance with company and industry standards and best practices for data security and regulatory requirements
  • Stay updated on emerging data engineering technologies and data infrastructures; evaluate their potential impact and application in our systems and processes
  • Provide technical leadership and mentorship to junior data engineers. Forster a culture of knowledge sharing and continuous learning
  • Participate in an on-call rotation (but rarely paged/alerted) and be able to occasionally work with flexible hours
Aufrufe: 1
Veröffentlichtvor 1 Tag
Läuft abin 18 Tagen
ArbeitsmodusHybrid
Quelle
Logo
Logo

Ähnliche Jobs, die für Sie von Interesse sein könnten

Basierend auf "Data Engineer - Site Reliability"