Amazon Nova Act speeds up software delivery through agentic quality assurance automation in Part 3.

Quality assurance platform QA Studio adds applications and team access controls

Meta description: QA Studio now helps organizations scale agentic QA by grouping tests into applications and controlling access with teams, roles, and shared permissions.

Tags: quality assurance automation, Amazon Nova Act, AWS, CloudWatch, team-based access

Featured image: Abstract illustration of applications, test suites, role badges, and metrics charts, with no real people, logos, or text.

For quality assurance and platform teams, AWS has presented QA Studio, a reference solution built with Amazon Nova Act, as a way to scale agentic QA automation. The first post introduced QA Studio, where teams define tests in natural language and run them through AI-powered visual navigation. The second post added test suites and a command-line interface for batch regression testing in continuous integration and continuous delivery pipelines. This latest post covers applications, which organize tests and report health metrics, and team-based user management, which controls who can see and change them.

Why the bottleneck moves from tests to ownership

As automation adoption grows, a platform team stands up test automation, then a second team joins, then a dozen more. Soon hundreds of test cases belong to different products and squads, and everyone can see everyone else’s tests. No one can answer at a glance whether an application’s coverage is healthy right now. Test authoring stops being the bottleneck; organizing tests and controlling who can touch them becomes the work. The next question is ownership: who is accountable when coverage slips? Without a clear owner, health problems have no one to land on.

QA Studio answers with applications. An application represents the system under test: a website, a web app, or a service front end. It gives a collection of tests a home and an owner. Use cases, the individual tests from Part 1, and test suites, the regression collections from Part 2, belong to an application and inherit its ownership. Teams work inside owned applications instead of one flat list. Creating an application takes a name, a base URL, and optional environments such as development, staging, and production. The application page lets teams create use cases, add existing ones, group suites, and view health. The reference application detail page shows pass rate, executions, failures, use-case tabs, and test-suite tabs.

Health metrics show whether test programs are stable

Results over time tell teams whether an application’s development lifecycle is healthy, not only whether one build can release. QA Studio records every execution as Amazon CloudWatch custom metrics, dimensioned by application and environment.

The top of the application page gives three figures: pass rate over the selected window, total executions, and current failures. A health score from 0 to 100 starts from pass rate and subtracts a penalty for flaky tests, so a suite that flips between pass and fail does not look falsely perfect. Teams can scan a portfolio and separate stable applications from those needing attention.

The metrics tab shows trends. Teams choose a 7-day hourly window or a 30-day daily window, then chart executions, successes, failures, and average duration. Because metrics carry an environment dimension, they can filter one environment, such as staging versus production for the same tests.

Two views focus on time-consuming tests. The flaky tests view ranks use cases by how often they flipped between passing and failing over the last seven days, surfacing intermittent failures that erode trust in a suite before they turn into a production miss. The recent failures view lists the latest failed executions for triage, and each links to Amazon Nova Act trajectory logs and session recordings from Part 1, moving from failure to what the agent saw in one click. The metrics tab also shows pass rate, executions, average duration, a daily success-and-failure chart, failure hotspots, and flaky use case ranking.

Teams, roles, and shared access keep control practical

QA Studio organizes access around teams. An application belongs to exactly one team. Its members work with it; other users do not see it. Each member holds one of five cumulative roles:

• Admin: studio-wide access to every team, application, and user, including creating teams and assigning applications. It is a studio-wide role, not a per-team assignment.
• Team Lead: manages their own team members and creates applications for the team.
• Editor: creates and edits use cases and test suites in team applications.
• Executor: runs use cases and suites but does not modify them.
• Viewer: read-only access to team applications and results.

A Team Lead has everything an Editor has, and so on down to Viewer. Access to a use case or suite follows its application. If a member can work with the application, they can work with its tests within their role. This keeps permissions at application level instead of across thousands of tests.

An administrator sees the whole studio. The Users page lists every account with team and role, filterable by team and role. An admin can tell who holds Team Lead on which team and who carries Studio-wide Administrator rights. In the reference deployment, four users span two teams: two Team Leads, one Viewer, and one Studio-wide Administrator. The Teams page rolls the same data by team, showing member and application counts side by side, so an admin can see which teams are expanding before opening any one. The Applications list shows every application across teams with base URL, owning team, use-case count, and last execution status.

Teams also share work. A platform team might own an application that a feature team needs to run against without changing it. QA Studio supports view, execute, or full shared access. A ceiling rule applies: a user’s effective permission is the lower of their own role and the grant level. A Viewer given full access remains view-only.

A team lead sees My Team with four tabs: Members, Applications, Use Cases, and Shared Access. The Members tab lists people and allows inline role changes or adding a member. In the AWS Team example, the lead manages two members, one Team Lead and one Viewer. The Applications tab shows only team-owned applications with test-case counts and last-run times, keeping the lead focused on their own surface rather than the full studio inventory. The Shared Access tab records which other teams can reach the team’s applications and at what level. In the example, AWS Team granted AMZN Team view access, and an AMZN Team Editor still sees the application read-only because of the ceiling rule.

AWS services behind the control plane and metrics

QA Studio builds on the earlier serverless foundation and adds several AWS services. Amazon Cognito maps each team to groups, one per role. An AWS Lambda authorizer resolves caller groups into API scopes and team context on every request. Amazon DynamoDB stores teams, memberships, application ownership, and share grants in an existing single-table design. Amazon CloudWatch receives custom metrics for every execution in the QAStudio/Executions namespace, dimensioned by application and environment; the metrics API reads them for health views and charts. AWS Lambda and Amazon API Gateway enforce two access layers: the role’s scope and a resource check that confirms the caller’s team owns or has been shared with the application. Because access comes from Cognito groups, the OAuth client credentials flow from Part 2 fits the same model, letting a continuous integration pipeline authenticate as a machine identity with only needed scopes.

Deployment, impact, and what to try next

Applications, metrics, and team-based access are part of the QA Studio reference solution, deployable in an own AWS account with the AWS Cloud Development Kit, or AWS CDK. Test data, results, and metrics stay within account boundaries. After deployment, an administrator creates teams, assigns members and roles, and assigns applications to teams. Team health metrics populate as tests run. Instructions and configuration details are in the QA Studio GitHub repository and its documentation.

The value for readers is a path from individual agentic tests to governed automation across many teams without losing visibility or control. Applications provide homes for use cases and suites; health reporting uses pass rate, trend charts, a flaky-adjusted score, and failure triage built on CloudWatch; team-based management backed by Cognito keeps teams inside owned applications while sharing stays bounded by the ceiling rule. This builds on Part 1 natural-language authoring and Part 2 suites and continuous integration and continuous delivery integration. To try it, deploy the reference solution, create a first team and application, and run use cases to watch metrics form. More details on the agent are on the Amazon Nova Act product page.

Vinicius Pedroni is a Senior Solutions Architect at AWS for the Travel and Hospitality Industry, with focus on Edge Services and Generative AI. Vinicius is also passionate about assistin.