Senior Automation Test Engineer – AI Systems (f/m/x)

Technologies we use

About the project

Your responsibilities

• Design, build, and maintain evaluation frameworks for Large Language Models (LLMs) and custom agentic harnesses
• Develop automated scoring mechanisms, assertions, and test suites to measure agent accuracy, reasoning reliability, hallucination rates, and tool-use correctness
• Simulate complex, multi-step agent workflows to uncover edge cases in autonomous decision-making
• Collaborate with engineering teams to set up robust monitoring frameworks for data drift, concept drift, and model performance degradation in production
• Implement telemetry and alerting systems to track latency, token usage, cost efficiency, and output quality over time
• Own the end-to-end testing strategy across our entire stack, ensuring seamless integration between the Angular frontend, Node.js backend services, and Python-based AI microservices
• Write clean, maintainable, and scalable automated test scripts in Python and JavaScript/TypeScript
• Perform rigorous API testing, integration testing, and performance/load testing for high-throughput microservices
• Integrate AI evaluation and testing harnesses into our CI/CD pipelines to ensure continuous quality gate enforcement before deployment
• Define and track key quality metrics (KPIs) for software reliability and AI model behavior

Our requirements

• Minimum 5 years of experience in software testing or quality engineering, with at least 2 years focused on testing or evaluating AI/ML applications, LLM workflows, or agentic systems
• Proficiency in Python for backend/AI testing scripts and familiarity with JavaScript/TypeScript for Node.js and Angular testing ecosystems
• Experience with evaluation frameworks such as LangSmith, Langfuse, EvidentlyAI, Ragas, or MLflow, and the ability to select appropriate metrics for classification or generation-quality problems
• Familiarity with ML drift monitoring concepts, metrics tracking, and observability tools like Datadog or Grafana
• Strong system-level design skills and the ability to build production-grade test and evaluation infrastructure from the ground up
• Advanced level of English

Optional

• Understanding of agentic design patterns and their decomposition into testable units
• Experience working under the regulatory conditions of healthcare systems

What we offer

• Great Place to Work since 2015 - it’s thanks to feedback from our workers that we get this special title and constantly implement new ideas
• Employment stability - revenue of PLN 2.1BN, no debts, since 2006 on the market
• We share the profit with Workers - over PLN 76M has already been allocated for this aim since 2022
• Attractive benefits package - private healthcare, benefits cafeteria platform, car discounts and more
• Comfortable workplace – class A offices or remote work
• Dozens of fascinating projects for prestigious brands from all over the world – you can change them thanks to Job Changer application
• PLN 1 000 000 per year for your ideas - with this amount, we support the passions and voluntary actions of our workers
• Investment in your growth – meetups, webinars, training platform and technology blog – you choose
• Fantastic atmosphere created by all Sii Power People