Monitor Your Agent After Launch
Launching is the start of the work. Real customers ask things you did not predict, and your products, prices and procedures change. Achilles gives you a small set of pages that show what is happening, and this page turns them into a routine your team can follow in under an hour each week.
The pages you will use
| Page | Under | What it tells you |
|---|---|---|
| Overview | Build | Conversations today, this week and this month, messages, cost estimate, and handoffs waiting |
| Sessions | Operate | Every conversation, with a replay and a step-by-step timeline |
| Transfer Sessions | Operate | Conversations passed to a person, and waiting for action |
| Health | Operate | A score for each agent and whether it is ready to deploy |
| Evals | Evaluate | Your saved test questions, run against the live agent |
| Audit Log | Operate | What your team changed, and a summary of refusals and errors |
Conversation review
Open Sessions, filter by agent and recent days, and read a handful of conversations in full. The replay shows what the customer saw, and the timeline shows what the agent did. Look for:
- Questions the agent could not answer, which point to missing abilities or procedures.
- Wrong or awkward wording.
- Refusals that seemed unnecessary, which may mean a role is too tight.
- Anything that should have been escalated and was not.
Sample conversations from different roles and channels, not only the ones that were flagged.
Health score
The Health page runs a set of checks on each agent and gives a score out of 100 along with a clear yes or no on whether it is ready to deploy.
| Kind of check | Effect |
|---|---|
| Serious problems, such as a role pointing at an ability that does not exist | Block deployment until fixed |
| Advice, such as missing greeting, thin test coverage or no refusal tests | Lower the score but do not block |
Use the score as a prompt to tidy up, not as a target in itself. Warnings never stop a fix from going live, so the team can act quickly.
Evaluations
An evaluation is a saved question with what you expect to happen: a certain ability is used, the agent refuses, the reply contains a phrase, or the conversation is escalated. Run them from Evals under Evaluate.
Tip: whenever a review turns up a real problem, add a matching evaluation. Over time, this becomes a safety net that catches problems before customers do. Run the whole set before every change goes live, and remember to include role-specific cases, for example a check that a junior role is refused a manager-only lookup. See Test your agent.
Insights
Use the Overview for trends: volume, cost and handoffs. Watch for sudden rises in handoffs, refusals or errors after a change. The Audit Log summary shows refusals and errors together with the change history, which helps you connect a change to its effect.
A weekly routine
- Overview (5 minutes). Note volume, cost and handoffs waiting.
- Transfer Sessions (10 minutes). Make sure every handoff has been picked up, and note common reasons. See Human handoff.
- Sessions (20 minutes). Read around ten conversations, mixing channels and roles.
- Health and Evals (10 minutes). Check scores and run the evaluations.
- Audit Log (5 minutes). Confirm that changes were made by the people you expect.
- Decide (10 minutes). List improvements, assign owners, add evaluations for anything fixed.
Monthly, for compliance
- Sample conversations for each role, and confirm hidden values stayed hidden. See Personal data.
- Check that each change to your data had an approval recorded. See Approvals.
- Review the console member list. See Users and teams.
- Check your retention approach. See Audit trail.
Put the weekly review in a shared calendar with a named owner. Teams that review conversations regularly find that most of their improvements come from a few repeated patterns.
For a tour of every page, see Console overview. For help with unexpected behaviour, see Troubleshooting.