Today's task
Once an agent is in use, "it seems fine" is not good enough. Today you'll open the Monitor page for a published agent, learn what its outcome numbers mean, and download a week of sessions to find the questions it handles badly. This follows Prompts, instructions or knowledge?, where you learned which lever to adjust.
You'll need: a published Copilot Studio agent with a few real conversations. You must be the agent owner or have it shared with you. To read transcripts you also need the Bot Transcript Viewer security role, which only admins can grant.
Heads-up: Microsoft updates Copilot Studio often. If your screens look different, look for the same names. Chats in the test panel don't appear in Monitor, and new sessions can take up to an hour to show.
Steps
- Open Monitor. Open your agent and select Monitor on the top menu bar. Pick a date range with real traffic.
- Read the Overview. Daily and monthly active users only appear if the agent requires authentication under Settings > Security > Authentication. Note the numbers, but don't judge yet.
- Check Conversation outcomes. Under Effectiveness, find the chart showing Resolved, Escalated, Abandoned and Unengaged sessions. Select See details to see which topics led to each outcome. A session is Abandoned if someone engaged and then went quiet for 30 minutes without a resolution.
- Check Customer satisfaction and Reactions. Look at the satisfaction score, and the thumbs-up and thumbs-down counts. Reactions are on by default (except on the Microsoft Copilot channel), and you can add a disclaimer about how feedback is used under Settings > User feedback.
- Review the Use section. Look at Generated answer rate and quality and Knowledge source use. Unanswered questions and Poor quality answers usually point to missing or outdated knowledge.
- Download the sessions. Above the Overview card, select Download Sessions, then choose a day's row. You get a CSV with the outcome, the chat transcript and any user comments. If the option is missing, an admin may have turned off transcript downloads.
- Read ten transcripts. Filter the
SessionOutcomecolumn to Abandoned and Escalated. For each, decide: was it a prompt, instructions or knowledge problem? Write down the single change you'd make.
What to notice
- Numbers tell you where to look. Transcripts tell you why. An abandonment rate means little until you read the conversations behind it.
- Data doesn't last. The docs say session details and transcripts are available for the last 28 days, and downloaded transcripts cut each agent response to 512 characters. When SharePoint is the knowledge source, the answers are redacted in the file. Transcripts aren't written at all for Microsoft Copilot agents or Dataverse for Teams environments. Export what you need before it ages out.
- Some analytics are in preview. Sentiment and custom metrics are labeled preview in the docs, so don't base decisions on them alone.
Governance tip: Transcripts contain what real people typed, which may include personal or confidential details. Give the Bot Transcript Viewer role to as few people as possible, tell users their chats are reviewed, and delete downloaded files when you're done.