We are looking for an Automated QA Engineer to join our team who will focus on end-to-end quality assurance.Your responsibilities:● RAG Quality Evaluation: Design and curate datasets for RAG evaluation, develop and execute tests for RAG benchmarking, evaluating the accuracy and relevance of AI-generated content and information retrieval.● Data-Driven Validation: Use data analysis techniques to evaluate test outcomes, track AI performance trends, and detect regression or drift across releases.● Automation Test Solution Development: Design and implement automated test suites for both OutSystems front-end modules and backend microservices interacting with RaaS from the Galileo platform.● Regression Testing: Build and maintain automated regression pipelines to ensure continuous platform stability after deployments and LLM-model updates.● Integration Testing: Verify seamless communication between LUMIN and Galileo APIs, ensuring data integrity and correct processing across layers.● CI/CD Integration: Embed automated tests into CI/CD pipelines (GitLab CI preferred) to enable consistent, repeatable quality gates before releases.● Performance & Reliability Testing: Monitor and test system performance and reliability across environments, identifying performance bottlenecks or RAG degradation issues.We are looking for you, if you have:● Experience: 4+ years of professional experience in automated software testing, including experience with web, API, and backend systems. Strong understanding of regression and integration testing principles. Proven experience in AI-app testing, specifically with applications using LLMs and RAG.● Automation Skills: Hands-on experience with automated testing tools (e.g., Playwright, Selenium, or OutSystems test automation frameworks). Experience developing custom test scripts in Python or JavaScript/TypeScript.● AI Quality Testing: Experience testing LLM and RAG applications, including prompt validation, factuality testing, and output benchmarking.● Backend & API Testing: Proven experience validating RESTful APIs, microservices, and asynchronous workflows. Familiarity with API testing tools (e.g., Postman, pytest, REST Assured).● CI/CD & DevOps: Working knowledge of GitLab CI or equivalent systems (e.g., Github Actions), with test automation integrated into build pipelines.We offer:● Interesting and challenging projects● Flexible working hours● Friendly, non-corporate atmosphere● Stable working conditions (CoE or B2B)● Possibility for self-development and promotion in the company● Rich benefits package● Work model: remote or hybrid (2 days per week from the office)