Big Data Engineer

Key Responsibilities1. End-to-End Project Ownership

• Gather requirements directly from business users
• Perform business analysis and data analysis
• Design system architecture
• Develop solutions
• Support testing and validation
• Work with DevOps team for deployment
• Provide ongoing support after release2. AI/GenAI Solution Development
• Build applications leveraging LLMs (e.g., OpenAI APIs)
• Implement use cases such as:o Content summarisationo intelligent filtering and categorisation
• Evaluate when AI is appropriate vs traditional coding approaches3. Full Stack Development
• Work across the entire stack rather than specialised roles
• Handle:o Backend logico Data processingo Integration layers
• No separation into traditional roles (DB, UI, middleware)4. Architecture & Design Decision-Making
• Independently design and justify technical architecture
• Make decisions for greenfield projects
• Explain trade-offs and reasoning behind implementation choices5. Stakeholder Interaction
• Engage directly with:o Business userso Product owner
• Translate business needs into technical solutions
• Handle feedback and change requests directly6. Agile & Independent Working Model
• Work in a small pod (lean team)
• Handle projects independently with minimal supervision
• Participate in design discussions but own individual project delivery Technology Stack (Indicative)
• Primary Language: Python
• AI/ML: LLM integrations (OpenAI or similar)
• Data Platforms: Databricks, Delta Lake
• Cloud: Azure ecosystem
• Visualisation (optional): Power BI (not mandatory) Required Skills & Capabilities
• Core Skills
• Strong full stack development capability
• Solid understanding of system architecture and design
• Experience building end-to-end applicationsAI & Data Skills
• Experience working with GenAI / LLM-based solutions
• Ability to integrate AI into real-world use casesTech skills: Scala, Java, Spark, Shell, Azure, Sonar, docker hub, container registry, nexus, Postman, Azure CLI
• Programming Language: Scala, Java, Spark
• Tools: Gitlab, Sonar, docker hub, container registry, nexus, Postman, Azure CLI
• Build Scripting: Gradle, Maven, Shell
• Packaging Tool/Technology: Shell
• Cloud Technology: Azure
• Hosting: Experience with container management platforms like Kubernetes and databricks
• Data: Storage accounts, PostgreSQL, knowledge on Spark and Data Bricks API,
• Security: Azure AD (SAML-Token based, Single sign-on), Hashi corp Vault, Managed Identities (Service Principals)
• Monitoring: Log Analytics, Azure Metrics, Azure Monitor,
• Infrastructure as Code (IaC): ARM Test Automation:
• Test-Frameworks: Cucumber, Junit, Mockito
• Tools: Gitlab, Sonar, docker hub, container registry, nexus, Postman
• Build Scripting: Maven, Shell
• Service / API Test: I want to test your endpoints as thoroughly as possible
• End to End Test: of the integrated system (continuous integration can be a major help here)
• Solid knowledge of QA methodologies, test planning, system dependencies, and product integration phases