Data Engineer with DataBricks

ClientClient is a premier home improvement company, boasting a network of over 2,000 stores across eight European countries. You'll be part of an organization that values innovation and teamwork, where you contribute to the company's success, not just a single project.Project overviewWorking in cross functional team using Agile framework to design, build and maintain home improvement solution.TeamAs a Senior Data Engineer, you'll integrate into a dynamic data engineering team led by an experienced Data Lead. Your new team is a blend of talented Data Engineers, Data Designers, meticulous QA experts, and an agile Scrum Master, all dedicated to building exceptional data solutions.Position overviewData Engineer will offer engineering support and deliver data insights through the collection, analysis, and summarization of development and service issues. Your expertise will help steer the direction of our data strategies and ensure the highest quality in our technology offerings.ResponsibilitiesDesign, build, and maintain ETL processes, transforming raw data into actionable insightsManage and optimize large-scale data pipelines, ensuring data quality and integrityDevelop and optimize SQL queries for efficient data extraction, transformation, and analysisCollaborate with cross-functional teams to understand data requirements and implement effective data solutionsUtilize Python and PySpark for data engineering tasks, including scripting and automationLeverage Azure Databricks for advanced data processing and analyticsMonitor, troubleshoot, and optimize data workflows to ensure high performance and reliabilityRequirementsProven experience as a Data Engineer in a professional, production environmentStrong programming skills in Python and solid understanding of ETL processesAdvanced proficiency in SQL and experience working with large datasetsHands-on experience with Azure Databricks and Azure Data Factory (ADF)Solid understanding of data modeling, data warehousing concepts, and performance tuningStrong expertise in Databricks, including Delta Tables and Delta Live TablesAdvanced programming skills in Python and PySparkExperience with Unity CatalogProficiency in dbt (Data Build Tool)Solid knowledge of Medallion ArchitectureNice to haveExperience with data governance and data cataloging toolsFamiliarity with CI/CD practices and version control systems (Git)Understanding of data security and privacy best practicesExperience with additional cloud services or platforms