Product demo

    Agentic Quality Gate

    An AI coding agent has implemented a ticket. The tests are green and the pull request looks ready. Run the gate and see what the pipeline could not tell you.

    A worked example using a fictional payments codebase. No external repository is analysed, and the analysis below is prepared demonstration data.

    Change received

    payments-platform · PR #482 · a3f19c2

    PAY-1842 Prevent purchases when the customer has insufficient balance

    A customer must not be able to complete a purchase if their available account balance is lower than the total transaction amount.

    Reported by the developer agent

    • Added balance validation to checkout UI
    • Added insufficient balance error message
    • Added automated checkout tests
    Developer agent

    Complete

    CI tests

    18/18 passed

    Build

    Successful

    Pull request

    Ready for review

    Everything above says this change is ready.

    The Quality Gate reads the requirement, the acceptance criteria, the implementation and the tests, and decides whether they agree with each other.

    What this demonstrates

    An AI coding agent implemented a ticket, the suite passed, and the pull request looked ready to merge. The gate is the check that runs at that moment, before the merge, and it asks a different question from the one CI asks.

    Green tests are evidence about the code, not about the requirement

    A test suite tells you the code does what its tests say it does. It cannot tell you whether that is what the ticket asked for.

    When an agent misreads a requirement it does not only write the wrong implementation. It writes tests that encode the same misreading. The suite then passes because it is internally consistent, and consistency is not correctness. CI measures the first and is silent about the second.

    What the gate checks

    Each acceptance criterion on the ticket is traced to the code that implements it and the test that asserts it. Three shapes of failure come out of that: criteria with no implementation, criteria implemented but never asserted, and tests that assert the behaviour the code happens to have rather than the behaviour the ticket specified.

    The last is the one review misses most often, because nothing about it looks wrong. The file is tested. The test passes. The assertion simply describes the bug.

    Where it runs

    On the pull request, alongside CI rather than instead of it. CI answers whether a change is mechanically safe to merge. The gate answers whether it is the change that was asked for. A change can pass either and fail the other.

    Common questions

    Does this replace our test suite?
    No, it reads it. The inputs are the requirement, the implementation and the existing tests, and the output is where those three disagree. A codebase with few tests gives the gate less to work with, not more.
    What happens when a requirement is genuinely ambiguous?
    It is reported as ambiguous rather than resolved. An agent quietly guessing at an ambiguous requirement is the failure this is built to catch, so guessing on its behalf would defeat the point.
    Is this specific to AI-generated code?
    The failure mode is not new, since a person can misread a ticket too. It matters more with agents because they produce far more change per unit of review time, so the volume of code that no human has read closely goes up sharply.