Test Your Agent
Testing is how you turn "it seems to work" into "we are confident". You have two tools. Playground is for trying conversations by hand. Evaluations is for saving test cases that you can run again and again.
Playground: try it like a customer
- Open Playground from the console menu.
- Pick the agent and, if you want, the role to test as, such as branch manager or customer.
- Chat. Use real phrases, including typos, mixed languages and vague requests.
- Look at what the agent did. You can see which lookups ran and what they returned.
What to try
- Happy paths. "Show my renewals this week." "What is my claim status?"
- Missing details. Ask without giving a policy number.
- Things it must refuse. "Show me everyone's policies." Another customer's details.
- Actions. Ask it to send a reminder and check that it asks for approval first. See Approvals.
- Handoff. Say "This is a complaint, I want a manager." Check that it passes you to a person. See Human handoff.
- Off-topic and pushy requests. Ask for legal advice, or try to argue it out of its rules.
- Each role. Test as each type of user.
Evaluations: save your tests
When a conversation works well, save it as a test case. A test case has a question and the behaviour you expect.
| You expect | Example |
|---|---|
| A particular lookup is used | "Show my renewals" uses the renewal lookup. |
| A refusal | A request for another customer's data is declined. |
| Certain words in the reply | The reply mentions the due date. |
| A handoff | A complaint goes to a person. |
Test cases run as the least-privileged user by default. A case that passes at the lowest level of access works for everyone. When a case needs more access, such as a manager report, choose that role for the case.
Run all your test cases with one click under Evals. Run them again after every change, so you notice if something that worked has broken.
Agent Builder creates starter test cases from your SOP. Add your own for the situations you worry about most. See Agent Builder.
The health score
Under Health, each agent has a score from 0 to 100. It reflects how many checks pass, such as whether every ability has a clear description, whether roles are complete, and whether safety rules are present.
| Finding | Effect |
|---|---|
| Blocking problem | Deploy is locked until you fix it. |
| Warning | Lowers the score, but does not block deployment. |
A score tells you about set-up quality. Your test cases tell you about behaviour. You need both.
When is it ready to launch?
Use this checklist.
- All test cases pass.
- No blocking issues in the health score, and warnings are understood.
- You tested every role, including refusals.
- Actions ask for approval and handoff works.
- Sensitive details are hidden as intended. See Personal data.
- Someone outside the build team tried it.
- Compliance or risk has seen the results.
After launch
Keep testing. Review real conversations in Sessions, add new test cases from what you find, and watch results over time. See Monitor.
Ready? Continue to Website widget or Messaging channels.