AI Platform Senior Engineer, QA

About Accrete Accrete is an agentic managed services company: an AI workforce that does high-stakes, judgement-heavy work for government and enterprise clients — at the economics of software, not labor. At the core is Accrete’s Knowledge Engine, a dynamic context graph that captures an organization’s tacit knowledge, resolves data across silos, and builds a living, auditable ground truth. Expert agents reason against that ground truth to act on complex, high-stakes decisions — not just answer questions about them. From national security to commercial operations, Accrete delivers the work on one platform, with unlimited expert agents and expert judgement. About the Role: We are seeking an AI Platform Senior Engineer, QA – Agentic AI & Automation to lead quality assurance for our agent-driven platforms and AI models. This role focuses on validating intelligent agents, non-deterministic AI outputs, and cross-product workflows using advanced automation and AI evaluation techniques. You will play a key role in building our Enterprise AI Trust & Validation Framework , ensuring reliability, transparency, and scalability of AI systems. Key Responsibilities Agentic AI & System Testing Test multi-agent workflows, orchestration layers, and cross-domain intelligence systems Validate AI-generated narratives, themes, and analytical outputs. Perform groundedness, coherence, hallucination, and stability testing. Test conversational AI and chat-based agent systems. Evaluate consistency across multiple runs and detect output drift. Automation & Framework Development Design and maintain automation frameworks using Python, JavaScript/TypeScript, Pytest, and Playwright . Build E2E automation for UI, APIs, and agent pipelines. Develop reusable test utilities for AI validation and regression testing. Integrate automation with CI/CD pipelines. AI Model & Evaluation Testing Validate in-house AI/ML models and inference pipelines. Implement LLM-as-Judge and semantic similarity-based validation. Perform statistical and comparative testing for model outputs. Support testing of models deployed on AWS. A PI & Performance Engineering Perform API testing using Postman and automated scripts. Design and execute load and stress testing. Monitor system performance under AI-heavy workloads. Identify bottlenecks and scalability risks. Quality Strategy & Collaboration Analyze business requirements, user stories, and AI specifications. Design test strategies for agentic and AI-driven systems. Work closely with Engineering, Product, and Data Science teams. Track defects, quality metrics, and release readiness. Drive continuous improvement in QA processes Required Qualifications 6–8+ years of experience in QA, Automation, or Quality Engineering. Strong proficiency in Python, JavaScript/TypeScript, Pytest, and Playwright . Experience testing AI systems, chatbots, or LLM-based platforms. Strong understanding of…