Senior QA AI Engineer

Accountabilities:

• Lead quality engineering and AI testing initiatives to ensure software solutions are reliable, scalable, secure, and production-ready across cloud-based platforms.
• Design, develop, and maintain automated test suites for Java-based backend services and REST APIs, supporting consistent quality across development and release cycles.
• Own testing strategies for AI-powered features, including large language model output validation, regression baselines, tool-call accuracy, context management, and end-to-end integration testing.
• Develop continuous integration evaluation frameworks for non-deterministic AI outputs using rubric-based scoring, automated prompt regression suites, and behavioral drift detection.
• Collaborate with product owners, software engineers, and machine learning engineers to establish testable requirements, review acceptance criteria, and maintain quality throughout agile delivery cycles.
• Perform database validation against PostgreSQL and Microsoft SQL Server to verify data integrity, transformation logic, and backend workflow accuracy.
• Conduct AI-focused security testing, including prompt injection assessments, adversarial input validation, jailbreak testing, and sensitive data leakage checks.
• Define and enforce synthetic or masked test-data standards for AI testing, ensuring confidential and regulated information is handled appropriately.
• Monitor and report quality, performance, and AI observability metrics, including token consumption, streaming latency, tool-call success rates, model error rates, and reliability indicators.
• Integrate automated testing into CI/CD pipelines and establish effective quality gates and deployment controls.
• Conduct test-code reviews, mentor quality engineering team members, perform exploratory testing, and continuously improve testing processes and methodologies.
• Support performance and load testing for cloud-native SaaS platforms hosted on AWS and Microsoft Azure, validating scalability, availability, and reliability.
• Maintain comprehensive test documentation, traceability matrices, defect analysis reports, and quality metrics to provide transparency and alignment across stakeholders.

Requirements:

• 5–7 years of experience in software quality engineering, with strong hands-on experience testing Java-based backend services and REST APIs.
• Proficiency with test automation frameworks such as Selenium, TestNG, JUnit, RestAssured, or Cucumber for UI and API testing.
• Strong knowledge of relational databases, particularly PostgreSQL and Microsoft SQL Server, with experience performing data validation and integrity testing.
• Hands-on experience testing AI features integrated with large language models, including output validation, semantic evaluation, hallucination detection, regression testing, and streaming API behavior.
• Familiarity with Model Context Protocol (MCP) and agentic AI testing patterns, including tool-use accuracy, context management, and validation of deterministic versus non-deterministic outputs.
• Experience conducting AI security testing, including prompt injection, jailbreak validation, adversarial testing, sensitive data leakage assessments, and reliability or performance testing.
• Strong understanding of software design principles, microservices architectures, and system-level test strategy development.
• Demonstrated ability to identify complex edge cases and translate technical requirements into comprehensive quality strategies.
• Strong understanding of AI technologies and experience applying AI tools to support innovation, operational efficiency, and responsible AI adoption.
• Knowledge of AI risk management, governance, and responsible AI practices.
• Experience working with cloud-based platforms, preferably AWS and Microsoft Azure.
• Strong collaboration and communication skills, with the ability to work effectively across product, engineering, machine learning, and quality teams.
• Experience mentoring engineers and contributing to continuous improvement in testing standards and processes.
• Bachelor’s or Master’s degree in Computer Science, Mathematics, Physics, or a related technical field.

Benefits:

• Full-time remote opportunity based in India.
• Opportunity to work on advanced AI, software quality, cloud, and automation initiatives.
• Exposure to large language models, agentic AI, AI security, and responsible AI practices.
• Opportunity to influence quality engineering strategy and establish modern testing standards.
• Collaboration with product, software engineering, machine learning, and other technical teams.
• Opportunity to mentor team members and contribute to the development of high-performing engineering practices.
• Experience working with cloud-native SaaS platforms across AWS and Microsoft Azure.
• Opportunity to contribute to technology that supports complex risk assessment, resilience, and informed decision-making.
• Inclusive environment that values curiosity, diverse perspectives, integrity, innovation, and continuous learning.