Lead AI Testing Engineer

We are looking for a Lead AI Testing Engineer to drive the quality strategy for AI-driven and data migration initiatives spanning knowledge graphs, ontologies and RAG-based LLM features. You will define and execute QA practices across data ingestion, graph layers, rule evaluation engines and AI/LLM components while building automation frameworks that scale evaluation across complex, data-heavy systems.

Responsibilities

• Design and automate evaluation of RAG-based and LLM-driven features including grounding, answer accuracy, determinism/reproducibility, precision, recall and hallucination rateBuild test harnesses to scale evaluation beyond human-in-the-loop processes
• Create data-driven test suites for underwriting rules and validate rule execution via SPARQL evaluator, arithmetic/threshold logic and multi-condition scenarios
• Verify ontology schema correctness and instance accuracy against source data, perform reasoner consistency checks and SHACL validation
• Define and drive end-to-end QA strategy covering data ingestion, ontology/graph layer, rule evaluation engine, AI/LLM layer and integration with underwriting solutions
• Establish quality gates, acceptance criteria and test coverage models
• Maintain automation test suites across knowledge graph, data ontology layer, backend services and frontend layers
• Validate end-to-end flows across ontology, rule engine and decision output, perform contract testing and cross-system data consistency validation
• Integrate test suites into CI/CD pipelines and define release quality gates
• Champion automation-first and shift-left quality practices
• Leverage agentic AI and Gen AI tooling in testing and framework developmentRequirements8+ years of experience in software development, testing automation and DevOps
• Proficiency in Python test automation using pytest or equivalent, scripting and AI/ML library integration
• Expertise in testing Knowledge Graph or Data Ontology solutions using OWL2, RDF and SPARQL
• Skills in SHACL validation and graph databases such as GraphDB, including ontology validation, entity/relationship integrity, reasoner consistency checks and graph query correctness
• Proven ability to define and drive QA strategy independently for complex data-heavy systems, including building automation frameworks and implementing quality gates
• Skills in data validation using SQL and graph query languages such as SPARQL and Cypher, along with understanding of deterministic vs probabilistic systems
• Background in backend and API testing (REST) covering data validation, integration and E2E testing
• Proficiency in frontend web application test automation using Playwright or equivalent Python-compatible framework
• Hands-on experience using coding agents and agentic development daily
• Demonstrated experience testing and evaluating RAG-based Gen AI / LLM applications including grounding, answer accuracy and hallucination/determinism checks
• Applied knowledge of Gen AI / LLM evaluation frameworks and metrics such as precision, recall, criteria recall and efficiency, with proven ability to automate Gen AI/RAG evaluation at scale
• Experience in Agile delivery using Git and Azure DevOps or JIRA
• English proficiency at B2 level or higherNice to have
• Skills in semantic search testing and evaluation
• Familiarity with Vector Database integration and retrieval validation
• Background in data science covering ML concepts, data pipelines or data engineering collaboration
• Proficiency in API tooling such as Postman and Bruno
• Experience with quality-gate implementation in CI/CD pipelines such as Azure DevOps or JenkinsWe offerWe gather like-minded people:

Engineering community of industry professionals

• Friendly team and enjoyable working environment
• Flexible schedule and opportunity to work remotely within Poland
• Chance to work abroad for up to 60 days annually
• Business-driven relocation opportunitiesWe provide growth opportunities:

Outstanding career roadmap

Leadership development, career advising, soft skills, and well-being programs

Certification (GCP, Azure, AWS)

Unlimited access to LinkedIn Learning, Get Abstract, Cloud Guru

English classesWe cover it all:

Stable income (Employment Contract or B2B)

Participation in the Employee Stock Purchase Plan

Benefits package (health insurance, multisport, shopping vouchers)

Strategically located offices featuring entertainment and relaxation zones, table tennis and football, free snacks, fantastic coffee, and more

• Referral bonuses
• Corporate, social and well-being events
• Please, note:

The set of bonuses might vary based on the role you apply for – specifics will be discussed with our recruiter during the general interview.

We will reach out to selected candidates exclusively.

EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.