RAG with Azure AI Search: give your Foundry agent your documents

Retrieval-augmented generation, or RAG, is how an agent answers from your documents instead of from what its model happened to learn: the question goes to a search index first, and the best passages go to the model with the question. In Microsoft Foundry, Azure AI Search is the index, and there are two ways to wire it to an agent. One is simple and generally available; the other reaches several sources at once and is partly in preview. This guide helps you choose, then sets up the simple one properly.

Choose a route

Microsoft's own guidance: use agentic retrieval for new RAG work when relevance matters most and queries are conversational; use classic, single-query retrieval when you need generally available features only or simplicity. This guide sets up the tool; the knowledge base route is at the end. (Copilot Studio agents add an Azure AI Search index as a knowledge source instead; our Power Platform and Azure overview covers that.)

Build the index

The tool needs a vector index. The quickest way is the Import and vectorize data wizard in the Azure portal, which chunks your documents, generates embeddings with a model you choose, and creates the index. Whichever way you build it, the index must have:

• at least one Edm.String field that is searchable and retrievable;
• at least one vector field, Collection(Edm.Single), that is searchable;
• a retrievable content field holding the text you want the agent to quote;
• a retrievable field with the source URL, and optionally a title, so citations can link somewhere.

Add semantic ranking on the search service if you can; the tool's default query type is vector_semantic_hybrid, which combines vector, keyword and semantic reranking. The other options are simple, vector, semantic and vector_simple_hybrid. The search service and the Foundry project must be in the same tenant.

Connect the search service to the project

Use the project's managed identity rather than an admin key. It's the only option that works with private networking, and it means no key to leak or rotate.

1. On the search service in the Azure portal, copy the endpoint from Overview (https://<name>.search.windows.net) and, under Settings > Keys, select Both so role-based access is enabled.
2. On the Foundry resource, open Identity and copy the Object (principal) ID of its system-assigned managed identity.
3. Back on the search service, Access control (IAM) > Add role assignment: give that identity Search Index Data Contributor and Search Service Contributor. Use Check access to confirm both appear.
4. In the Foundry portal, create a project connection to Azure AI Search with the endpoint and managed-identity authentication. The Azure CLI equivalent is az cognitiveservices account project connection create; the old az ml commands don't work with Foundry projects and fail with "Workspace not found".

Note the connection's ID. The tool refers to the connection, not the service.

Add the tool to the agent

Add Azure AI Search to the agent, in the portal under the agent's tools or in code as part of its definition, with these settings:

Then tell the agent to cite. Citations come back as url_citation annotations on the response, but only if the instructions ask for them. Microsoft's template for a knowledge-grounded agent is a good start:

You are a helpful assistant.
Use the search tool to answer user questions.
If the search results don't contain the answer, respond with "I don't know".
When you use information from the search results, include citations to the retrieved sources.

An agent can have one Azure AI Search tool, and that tool targets one index. If several agents need the same index, put the tool in a toolbox, which shares it through one managed endpoint with one set of credentials.

Things that catch people out

The Foundry IQ route

When one index isn't enough, create a knowledge base in Azure AI Search with one or more knowledge sources, and connect it to the agent as an MCP tool through a project connection of category RemoteTool that uses the project's managed identity. The agent is then allowed exactly one MCP tool, knowledge_base_retrieve. Permissions: the project's managed identity needs Search Index Data Reader on the search service (plus Data Contributor if the agent writes to indexes); if the knowledge base uses a model for query planning or answer synthesis, the search service's identity needs Cognitive Services User on the Foundry resource; the person creating the connection needs Foundry Project Manager. For permission-aware answers, forward each user's token in the x-ms-query-source-authorization header so protected sources filter results per user. Agentic retrieval adds latency in exchange for handling multi-part and conversational questions, and the SDK and API versions it needs are preview, so check what's generally available before you build production on it.

Sources

• Connect an Azure AI Search index to Foundry agents, Microsoft Learn
• Retrieval-augmented generation (RAG) in Azure AI Search, Microsoft Learn
• Agentic retrieval in Azure AI Search, Microsoft Learn
• What is Foundry IQ?, Microsoft Learn
• Connect a Foundry IQ knowledge base to Foundry Agent Service, Microsoft Learn
• Quickstart: Vectorize text and images using the Azure portal, Microsoft Learn