QA Engineer (AI + backend)

Engagement: We are partnering with a US-based health-tech company on the takeover of a production AI-powered mobile coaching platform. Real users, real production, real releases — not a greenfield project.

Role summary: Own quality on the two hardest surfaces of a production AI coaching platform:

the LLM agents that answer differently every run, and the API layer everything else depends on.

As AI & Backend QA Engineer you own the quality of what the model says and what the APIs return. On the AI side that means building the evaluation practice: golden sets, quality metrics, and regression detection that catch a degraded answer before a user does. On the backend side it means API test suites and consumer-driven contract testing with Pact — agreed with the client's US engineering team as a standing requirement for this role, not an optional technique.

The defining skill is localization: when an answer is wrong, you can tell whether the problem sits in the model, in the service code, or in the input data — and you bring the evidence, not a guess.

What you'll be testing:

• AI core — a Python service with Lang

Graph agents producing streaming, non-deterministic coaching responses; this

• role's primary surface
• Backend APIs — a NestJS REST gateway behind Auth0, plus the services behind it
• Mobile app — React Native, shipping to iOS and Android app stores; AI features surface directly in the app that
• members use
• Data layer — MongoDB, Redis, and Postgres behind the services
• Process — Agile/Scrum, Jira for work tracking and defects, TestRail for test management
• You join the Polish QA function alongside our Manual QA Engineer and QA Automation Engineer, and work in a shared workflow with the client's US-based engineering and QA teams.

What you'll do:

• Build and run LLM evaluation. Curate datasets and golden sets, define response quality metrics, and catch quality regressions between prompt and model versions before they reach production
• Define acceptance criteria for non-deterministic output. Assertions on properties and contracts — structure, constraints, invariants — rather than exact wording
• Treat prompts as tested artifacts. Change a prompt, measure the effect of that change; probe systematically for hallucinations and prompt injection
• Own API testing. REST endpoints, JSON payloads, auth token flows, error handling — driven through Postman and Swagger/OpenAPI specs
• Implement contract testing with Pact. Consumer-driven contracts between the mobile app, the gateway, and the backend services, verified on every change
• Test with security in mind. Authorisation boundaries, token misuse, malformed input — the API-level abuse cases nobody writes a happy-path test for
• Localise failures across the stack. Logs, traces, and database queries to pin a defect to the model, the service code, or the input data
• Help wire evaluations into CI/CD. Evaluation runs in the pipeline and quality gates on pull requests, so a bad prompt change cannot merge quietly
• Keep the record straight. Results mapped to test cases in TestRail, defects logged and driven to closure in Jira, findings written up so the team can act on them

Must-have skills:

• 3+ years in technical QA or test automation, including API testing on production systems
• Python — writing and maintaining test code, pytest, fixtures
• Strong HTTP and API fundamentals — REST, JSON, status codes, headers, auth tokens
• Contract testing with Pact — consumer-driven contracts in practice; this is a hard requirement for the role, agreed with the client's engineering team
• Security-aware API testing — authorisation boundaries, token handling, malformed and hostile input
• Comfort with non-determinism — defining acceptance criteria when there is no single correct output: assertions on properties and contracts rather than exact wording
• LLM evaluation — datasets and golden sets, response quality metrics, catching quality regressions between prompt and model versions
• Prompt engineering — the ability to change a prompt and measure the effect of that change; awareness of hallucinations and prompt injection
• Backend savvy — HTTP and REST, JSON, tokens, logs, database queries; you can tell whether a problem sits in the model, in the service code, or in the input data
• Postman and Swagger/OpenAPI as daily tools
• Git in a team workflow — branches, pull requests, reviews
• TestRail for test-case management and run tracking
• Mobile product context (Android/iOS) — comfortable testing features that ship in a mobile app, including apprelease timing and store builds
• Strong English — you work in a shared workflow with US colleagues
• Must work hours that overlap US Mountain Time afternoons (~14:00–18:00 PL time)

Nice-to-haves:

• Langfuse or equivalent (LangSmith, Phoenix) — working with traces and attaching evaluations to them
• Lang

Graph — testing an agent as a state graph, both a single node and the full flow

• CI/CD — running evaluations in a pipeline and quality gates on pull requests
• Python with Appium — mobile automation exposure alongside the API work
• Testing other AI or LLM-driven features in production
• Experience taking over quality ownership of someone else's product

Offer:

• Location: 100% remote
• Engagement: B2B contract
• Duration: Initial 3-month contract, expected to extend
• Rate: upt to 160 PLN/h
• Start: asap