Python & Apache Spark Engineer - Data Processing Specialist

Technologies we use

About the project

This is how we organize our work

This is how we work

Your responsibilities

• Design, develop, and maintain robust, scalable, and production-ready Apache Spark applications.
• Develop efficient data processing, transformation, and analysis workflows using Python and Spark.
• Build solutions capable of handling large-scale datasets while maintaining high performance and reliability.
• Work with columnar data formats, particularly Parquet, and implement data solutions using Delta Tables.
• Develop clean, maintainable, testable, and efficient code in accordance with established engineering standards.
• Optimize Spark applications, data processing pipelines, and code for performance, scalability, and efficient resource utilization.
• Analyze and resolve technical issues affecting application reliability, performance, and data processing.
• Contribute to the design and implementation of scalable microservices-based solutions.
• Define, promote, and enforce software engineering best practices and coding standards across the project.
• Conduct detailed code reviews, ensuring code quality, maintainability, security, performance, and compliance with project standards.
• Identify potential technical risks and proactively propose improvements.
• Support the team in solving complex technical and architectural problems.
• Provide technical guidance to other team members and help establish consistent development practices.
• Contribute to continuous improvement of development processes, workflows, and engineering standards.
• Work both independently and collaboratively within a distributed, cross-functional team.
• Cooperate closely with software engineers, data specialists, architects, project managers, and business stakeholders.
• Communicate clearly with team members, coordinate day-to-day technical activities, and help ensure smooth delivery.
• Maintain regular communication with the client and participate in technical discussions and project meetings.
• Translate business and technical requirements into practical and scalable software solutions.
• Take ownership of assigned tasks and proactively communicate progress, risks, dependencies, and potential blockers.
• Create and maintain comprehensive technical documentation covering code, data processing workflows, architecture, processes, and operational procedures.
• Ensure developed solutions are maintainable and understandable for other team members.
• Support the delivery of high-quality software solutions within agreed timelines.
• Balance standardized engineering processes with a pragmatic and flexible approach to project-specific requirements.

Our requirements

• 5+ years of hands-on experience in software development, preferably in data-intensive or enterprise environments.
• Strong commercial experience with Apache Spark and large-scale data processing.
• Practical experience with one or more modern Spark-based platforms, such as:
• Databricks
• Azure Synapse
• Microsoft Fabric
• Strong programming skills in Python.
• Experience with data manipulation and processing using Pandas, Polars, or similar libraries.
• Strong understanding of Parquet and other columnar data storage formats.
• Practical experience working with Delta Tables / Delta Lake.
• Proven experience designing and implementing data processing, transformation, and analytical workflows.
• Solid understanding of microservices architecture and its implementation in scalable enterprise systems.
• Experience writing clean, maintainable, efficient, and production-ready code.
• Understanding of software development best practices, coding standards, testing, and code review processes.
• Strong analytical and problem-solving capabilities.
• Detail-oriented approach with the ability to identify potential technical issues and bottlenecks.
• Practical and pragmatic mindset — able to balance standardized processes with the flexibility required to achieve project objectives.
• Strong organizational and time-management skills.
• Ability to independently manage workload, prioritize tasks, and meet tight deadlines.
• Excellent communication skills and the ability to explain technical topics clearly to both technical and non-technical stakeholders.
• Comfortable working independently while also being an active member of a collaborative, international team.
• Willingness to take ownership and proactively drive technical topics forward.

Optional

• Experience with Microsoft Azure or other major cloud platforms.
• Practical knowledge of Azure services, particularly:
• Azure Service Bus
• Azure Data Lake
• Azure Blob Storage
• Azure Redis
• Other Azure SaaS/PaaS services
• Experience building APIs and backend services with FastAPI.
• Hands-on experience with Docker.
• Experience with Kubernetes and container orchestration.
• Familiarity with modern AI/GenAI technologies, including LangChain, LangGraph, RAG pipelines, or LLM-based applications.
• Experience with Azure Synapse Analytics or Microsoft Fabric.
• Experience working in enterprise environments with complex data processing requirements.