Requirements

Expected technologies

Azure DevOps

Azure Databricks

Our requirements

• Bachelor’s degree in computer science, data management, information systems, information science or a related field; advanced degree in computer science, data management, information systems, information science or a related field preferred.
• 3+ years in data engineering building production data pipelines (batch and/or streaming) with Spark on cloud.
• 2+ years hands-on Azure Databricks (PySpark/Scala, Spark SQL, Delta Lake) including:
• Delta Lake operations (MERGE/CDC, OPTIMIZE/Z-ORDER, VACUUM, partitioning, schema evolution).
• Unity Catalog (RBAC, permissions, lineage, data masking/row-level access).
• Databricks Jobs/Workflows or Delta Live Tables.
• Azure Data Factory for orchestration (pipelines, triggers, parameterization, IRs) and integration with ADLS Gen2, Key Vault.
• Strong SQL across large datasets; performance tuning (joins, partitions, file sizing).
• Data quality at scale (e.g., Great Expectations/Deequ), monitoring and alerting; debug/backfill playbooks.
• DevOps for data: Git branching, code reviews, unit/integration testing (pytest/dbx), CI/CD (Azure DevOps/GitHub Actions).
• Infrastructure as Code (Terraform or Bicep) for Databricks workspaces, cluster policies, ADF, storage.
• Observability & cost control: Azure Monitor/Log Analytics; cluster sizing, autoscaling, Photon; cost/perf trade-offs.
• Proven experience collaborating with cross-functional stakeholders (analytics, data governance, product, security) to ship and support data products.

Your responsibilities

• Responsible for working with the data management, data science, decision science, and technology teams to address supply chain data needs in demand and supply planning, replenishment, pricing, and optimization
• Develops/refines the data requirements, designs/develops data deliverables, and optimizes data pipelines in non-production and production environments
• Designs, builds, and manages/monitors data pipelines for data structures encompassing data transformation, data models, schemas, metadata, and workload management. The ability to work with both IT and business
• Integrates analytics and data science output into business processes and workflows
• Builds and optimizes data pipelines, pipeline architectures, and integrated datasets. These should include ETL/ELT, data replication/CI-CD, API design, and access
• Works with and optimizes existing ETL processes and data integration and preparation flows and help move them to production
• Works with popular data discovery, analytics, and BI and AI tools in semantic-layer data discovery
• Adepts in agile methodologies and capable of applying DevOps and DataOps principles to data pipelines to improve communication, integration, reuse, and automation of data flows between data managers and data consumers across the organization
• Implements Agentic AI capability to drive efficiency and opportunity