AI Test Engineer

Job title: AI Test Engineer

 

Location: Pan India

 Work Mode: Remote

 Candidate Specification:  3+ years overall, Relevant – 3+ years

 Candidate Location: Pan India

 

Skills:  MCP, RAG, agentic AI, LangSmith

 

Candidate Industry: IT

 Education and Certification: Any Graduation

 

Job Description

Must have 3+ years in software QA/testing, with hands-on senior-level experience evaluating LLM-based or agentic AI systems (not just traditional QA)

Direct experience with eval frameworks/tools — LangSmith strongly preferred; also acceptable: Ragas, DeepEval, PromptFoo, Braintrust, Langfuse

Strong understanding of RAG architecture and how to independently evaluate retrieval quality vs. generation quality

Practical familiarity with MCP (Model Context Protocol) or comparable agent tool calling frameworks

Strong Python skills for building test harnesses, scoring pipelines, and automation

Experience designing evaluation approaches for non-deterministic systems (probabilistic scoring, not binary pass/fail)

Comfortable working directly with client engineering teams — strong communication, able to explain quality risk to both technical and business stakeholders