Your first agent: instructions, knowledge and testing

Build a Copilot Studio agent that answers questions from your own content, and test it properly before anyone else sees it. Instructions that work, knowledge that's scoped, and a repeatable test set.

Copilot StudioFix a problem12 min read

19 sources · Updated 13 Oct 2026Posted by the Maker DeskHow we write guides

Note

As of September 2026; steps 1, 5 and 6 were updated in October 2026. Copilot Studio changes often. This tutorial describes agents built with the standard harness, the long-standing building experience. Microsoft is also rolling out a new experience built on the GitHub Copilot harness, where some screens and names differ. Check Microsoft Learn for the version you see.

It takes minutes to create an agent in Copilot Studio and much longer to make one people can trust. The difference is almost never the model. It's three things you control:

Clear instructions: what the agent is for, and how it should behave.
Scoped knowledge: where its answers come from.
Real testing: before anyone else sees it.

This tutorial builds a simple internal agent, an IT help agent that answers questions from an IT support SharePoint site, and covers each of the three properly.

Step 1: create the agent

Sign in to Copilot Studio and choose the environment your organisation uses for agents. If New experience is switched on, turn it off first; that brings up the Home page for standard-harness agents. Then either:

Describe it in your own words (up to 1,024 characters). Copilot Studio drafts a name, description and instructions, and suggests knowledge, tools, triggers and channels. You can add or dismiss each suggestion, and edit the name, description and instructions later on the Overview page.
Create a blank agent and write everything yourself.

Describing is quicker; blank gives you more control. Either way, the result lands on the agent's Overview page, where you'll do the rest.

Give the agent a name and description that say exactly what it does, for example: "Answers questions about laptops, accounts and Wi-Fi from the IT support site." Copilot Studio's orchestrator uses names and descriptions to decide what to use, so vague descriptions lead to vague behaviour.

Step 2: write instructions that work

Instructions tell the agent how to behave: its role, its tone, what it should do, and what to do when it doesn't know. Microsoft's guidance for writing them comes down to a few habits:

Structure them with Markdown: headings, bullet lists, and numbered steps only where order matters.
Say what to do, with precise verbs, such as "ask", "search" or "use". Don't only list what to avoid.
Name the knowledge and tools the agent should use. In the instructions editor, type / to insert a reference to a specific knowledge source, tool, topic or variable.
Specify tone, length and format. If you don't, the model guesses, and guesses vary.
Define your organisation's terms and acronyms.

Here's a starting point for the IT help agent:

text

# Role
You help employees with IT questions about laptops, accounts, Wi-Fi and printing.

# How to answer
- Use only the IT Support SharePoint site to answer.
- Keep answers short: a one-line summary, then up to five numbered steps.
- Include a link to the page you used.
- Use a friendly, plain tone. No jargon without explaining it.

# When you can't answer
- If the site doesn't cover the question, say so, and give the IT Service Desk
  contact from the "Contact us" page.
- Never guess at passwords, account names or security settings.

# Terms
- "SSPR" means self-service password reset.

Tip

"When you can't answer" matters more than it looks. An agent without it tends to fill gaps with plausible-sounding text. An agent with it says "I don't know" and points to a person, which is what builds trust.

Step 3: add knowledge, and keep it narrow

On the Overview page, select Add knowledge and choose SharePoint. Enter the URL of the IT support site.

Two things to know about SharePoint knowledge:

The agent answers with the user's permissions. When someone asks a question, it searches only content that person can open. Keep the default sign-in setting, Authenticate with Microsoft, so it knows who they are.
Only some content is read. The agent reads modern SharePoint pages and common documents, such as Word, PowerPoint and PDF. It doesn't use classic ASPX pages or modern pages built with SPFx components. New or changed content appears only after SharePoint's search index catches up.

Leave Web Search and general knowledge off for now. When they're on, the agent can answer from the public web or from the model's own training when your content doesn't cover a question. That makes answers richer but harder to verify. Turn them on later, deliberately, if you decide you need them.

Step 4: test in the test pane

Select Test to open the Test your agent pane, and ask the questions real users will ask, in the way they'll ask them:

"My laptop won't connect to the office Wi-Fi."
"how do i reset my password"
"Can I install software myself?"
Something the site doesn't cover, such as "What's the cafeteria menu?"

For each answer, check three things:

Is it correct? Compare it with the source page.
Did it use the right source? The citation should point to the page you'd expect.
Did it follow the instructions? Look at length, format, the link, and the "I don't know" behaviour.

The activity map shows the plan the agent made for each question, including which knowledge or tools it chose and how long each step took. When an answer is wrong, the map usually shows why: the wrong source, or no source at all. Use Reset to start a fresh conversation after you change instructions.

Step 5: turn your questions into a test set

Testing in the chat pane checks one conversation at a time, and it's hard to repeat exactly. Agent evaluation runs a whole set of test cases at once, and you can run the same set again after every change.

Where to start. Open the agent's Evaluation page, select New evaluation, and choose the kind of test set. You can also select the Evaluate icon in the test pane to start from your latest test chat.

Single responses: each test case is one question, asked on its own. A set holds up to 100. Start here for a question-and-answer agent like this one.
Conversation: each test case is a back-and-forth of up to 12 messages, and a set holds up to 20. It suits agents that must keep context over several turns; add one later if you need it.

Where the questions come from. For a single-response set, you can:

write them yourself, or import a CSV or text file with a Question column and an optional Expected response column (up to 100 questions, each up to 1,000 characters);
reuse the questions from your latest test chat;
have Copilot Studio generate them. A quick set gives you 10 questions based on the agent's description, instructions and capabilities. A full set draws on the agent's knowledge or topics.

Generating from knowledge works with text, Word, Excel and PDF files and individual SharePoint files, each up to 5 MB. SharePoint folders aren't supported, and a whole site isn't on Microsoft's list. For this agent, your own real questions are the better start.

Choose how answers are scored. General quality needs nothing from you. Most other test methods need an expected answer or keywords for every question, and a question without them comes back Invalid for that method.

Test method	What it checks	For a first agent
General quality	Relevance, groundedness, completeness, and whether the agent attempted an answer. On by default; needs no expected answers	Yes
Compare meaning	Whether the answer means the same as your expected answer, however it's worded. Default pass score: 50	Yes, once you've written expected answers
Keyword match	Whether the answer contains any, or all, of the words or phrases you list	Optional, for must-haves such as a page name
Custom	Your own instructions and labels, each label set to pass or fail	Yes, for questions the agent should decline (below)
Text similarity and Exact match	How closely, or exactly, the wording matches your expected answer	Rarely: they suit fixed wording such as codes, not answers that can be phrased several ways
Tool use	Whether the agent used the tools or topics you expected	Not yet: this agent has no tools
Content safety	Harmful content of a chosen type, against a severity level you set	Optional

Questions the agent should decline. General quality checks whether the agent attempted an answer, so a correct "I can't help with that" can be flagged for improvement. Microsoft's documentation suggests a Custom method that defines the expected refusal. A test set's methods score every question in it, so we keep out-of-scope questions in a second, small set: remove General quality from it and add a custom method like this.

text

Name: Declines out-of-scope questions

Evaluation instructions:
The user asked about something the IT support site doesn't cover.
Check whether the agent:
- says it can't answer that from the IT support site
- gives the IT Service Desk contact
- doesn't make up an answer

Label 1: Declined correctly
Result: Pass
Description: Says it can't help and points to the IT Service Desk.

Label 2: Answered anyway
Result: Fail
Description: Gives an answer or a guess instead of declining.

Label names accept only letters, numbers, spaces and a few symbols (hyphen, underscore, slash, ampersand, plus and full stop), so leave apostrophes out of them.

Run it, then run it again. Save the set and run it. A run takes a few minutes, and only one runs at a time. Each question comes back Pass, Fail, Invalid or Error, and the set gets a pass rate. Open a failed question to see the expected and actual answers, the reason for the result, and the activity map. After you change the agent, run the same set again and use Compare with to see which questions went from pass to fail.

Before you trust the scores:

It runs as you. An evaluation uses the signed-in maker's account and connections. SharePoint answers depend on who's asking, so the scores reflect what you can open. To test as someone else, sign out and sign in as that person. Generated questions can include content your account can see, and any maker with access to the agent can view its test sets.
Results are kept for 89 days. Export a run to CSV to keep it longer.
The sign-in setting matters. Evaluation supports agents set to Authenticate with Microsoft or No authentication, not Authenticate manually. If evaluations that use your sign-in won't run, ask whether an admin has turned off the Microsoft Copilot Studio connector in a data policy.

Tip

Save twenty to thirty real questions as your first test set, and run it before every publish. It's the fastest way to notice that a change to the instructions fixed one answer and broke three others.

Step 6: publish to a small group first

When the test set passes, select Publish. You can't publish on a trial licence. After the first publish, every later change stays in draft until you publish again.

Then choose where people will use it on the Channels tab, for example Teams and Microsoft Copilot:

Start small. Share it with a pilot group, not the whole organisation.
Expect an approval step for a wider release. To show the agent to everyone in your organisation, you submit it for an admin's approval. It appears in the Built by your org section of the Microsoft 365 Agent Store only after an admin approves it.

Checklist

The name and description say exactly what the agent does.
Instructions set the role, the format, and what to do when the agent can't answer.
Knowledge is limited to the sources the agent needs; web search and general knowledge are off unless chosen deliberately.
Answers were checked for correctness, source and format in the test pane.
A single-response test set of real questions runs before every publish, scored with General quality, and with Compare meaning once expected answers are written.
Questions the agent should decline sit in their own test set, scored with a Custom method.
The first release goes to a pilot group.

The steps follow Microsoft's documented behaviour for Copilot Studio agents using the standard harness, as of September 2026. Steps 1, 5 and 6 were checked again against Microsoft Learn in October 2026. The example instructions, the custom test method and the test approach are our own recommendations.

Sources

Did this fix it?

Comments

No comments yet. Share what worked for you.