LowCodeStacks™
Sign in

Cosmos DB 429 request rate too large: when it is fine, and how to find the hot partition

A 429 from Cosmos DB means one second's request units ran out. If 1 to 5 percent of requests get one and latency is fine, that's a full tank, not a fault, and the SDKs retry up to nine times before your app hears about it. Above that, the cause is total load, a hot partition, metadata calls or a transient error, each with its own fix. The metrics to open, the log query that names the partition key, and what autoscale does and doesn't do.

Data & storageFix a problem7 min read

Checked against Microsoft Learn · 7 sourcesPosted by the Maker DeskHow we write guides

Cosmos DB 429 "Request rate too large" is the service's way of saying that this second's budget of request units is spent; next second, the budget is back. That makes it unlike most errors: a small, steady rate of them is the sign of a container that's using what you pay for, and the SDKs, Data Factory and the bulk tools retry them silently. The error becomes a problem when it's sustained, when it hits one partition key while the rest of the container idles, when it comes from metadata calls that no amount of throughput fixes, or when it arrives on a container whose autoscale "should have" absorbed it. This guide takes Microsoft's own three-step diagnosis and adds the queries, the SDK defaults and the autoscale rules people trip on.

What a 429 is

Every operation costs request units; a container with provisioned throughput gets a fixed number of RU/s, or with autoscale a range up to a maximum. Spend more than that in one second and the extra requests get a 429 with an x-ms-retry-after-ms header saying how long to wait. The budget resets every second. The response carries one of four messages, and they aren't the same problem:

  1. "Request rate is large": the normal case, this guide's main subject.
  2. "The request didn't complete due to a high rate of metadata requests": a separate, system-reserved budget; see metadata 429s.
  3. "The request didn't complete due to a transient service error": retry; raising throughput does nothing.
  4. TXN_WAIT_FOR_TRANSACTION_END: a transactional batch is in flight on that partition key; retry.

Failed requests cost RUs too: a 400, 412 or 449 is charged, so a spike in conflicts or precondition failures shows up as throttling.

Step 1: how many, and does the app see them?

In the account's Insights, open Requests, then Total Requests by Status Code, and filter to the database and container. Compare the 429 count with the successes. Microsoft's threshold for a production workload is plain: 1 to 5 percent of requests with 429s, with acceptable end-to-end latency, is healthy; it means the provisioned RU/s are fully used. The caveat is that this assumes partitions are evenly loaded, which step 2 checks.

If the metrics show 429s and your application never logs any, that's the SDK. The .NET, Java, Node.js and Python SDKs catch a 429, wait the server-specified time, and retry; the default is 9 retries within a 30-second cumulative wait, after which the exception reaches your code. Data Factory and the bulk executor library retry the same way. Occasional 429s the SDK absorbs are fine; 429s that reach the application mean the retries ran out, which is sustained load.

Measure before you change anything: raising RU/s to make an occasional 429 disappear is paying for headroom you never use.

Step 2: the hot partition

Throughput is divided evenly between the physical partitions (partition key ranges). A container at 20,000 RU/s with four physical partitions gives each 5,000, and a logical partition key that draws most of the traffic exhausts its physical partition's share while the others sit idle. The symptom is 429s with total consumption well below what's provisioned.

In Insights, open Throughput, then Normalized RU Consumption (%) By PartitionKeyRangeID. Each range is one physical partition. One range consistently near 100 percent while the others sit at 30 or below is a hot partition.

To find which logical key, turn on diagnostic logs to Log Analytics for a limited period (it's billed by volume) and query the resource-specific table:

kusto
CDBPartitionKeyRUConsumption
| where TimeGenerated >= ago(24h)
| where DatabaseName == "<database>" and CollectionName == "<container>"
| where isnotempty(PartitionKey)
| summarize RUsPerSecond = sum(RequestCharge) by PartitionKey, OperationName, bin(TimeGenerated, 1s)
| order by RUsPerSecond desc

The top rows name the key. Learn's two examples are the ones everyone has:

  • Partitioned by date for a write-heavy stream: every write of the day lands in one logical partition. Use a high-cardinality key such as the device or item id, or a synthetic key that combines id and date.
  • Partitioned by tenant with one tenant far bigger than the rest. Give the large tenant its own container with a finer key, such as user id.

Changing the key means a new container and a copy; Microsoft's container-copy jobs do it with the data online. In the meantime, more RU/s raises every partition's share, so it's a stopgap, not a fix.

Step 3: which requests, and what they cost

With the hot partition ruled out, the question is what's spending the RUs. The same diagnostic logs tell you:

kusto
let throttled = CDBDataPlaneRequests
| where StatusCode == "429"
| project OperationName, TimeGenerated, ActivityId, RequestCharge;
CDBQueryRuntimeStatistics
| project QueryText, ActivityId, DatabaseName, CollectionName
| join kind=inner throttled on ActivityId
| summarize count(), avg(RequestCharge) by QueryText, OperationName
| order by count_ desc

Expensive queries without the partition key in the filter, large documents, and writes on containers that index every property are the usual answers. The capacity planner on Learn turns your volume, operation mix and document size into an RU/s estimate; for writes, trimming the indexing policy to the paths you query cuts the charge per write.

When you do raise throughput, the change is instant up to 10,000 RU/s per physical partition (number of partition key ranges times 10,000). Above that, Cosmos DB has to add partitions, which runs asynchronously and can take five to six hours.

Autoscale still returns 429s

Autoscale is not a promise of none. Two cases:

  • You exceeded the maximum. Autoscale with a 20,000 RU/s maximum throttles at 20,000, exactly as manual throughput would.
  • A hot partition. The maximum is split across physical partitions too, so one key can exhaust its partition's share of the maximum while the container as a whole is nowhere near it. Step 2 applies unchanged.

And the question that follows: "normalized consumption hit 100 percent, so why didn't it scale to the maximum?" Because autoscale scales to the maximum only when consumption is at 100 percent for a sustained period within its 5-second interval. A momentary spike scales to something above the previous level and below the maximum, which keeps the bill down, and the spike's extra requests were still served from that second's budget. If consumption sits at 100 percent and the container is permanently at its maximum, autoscale is costing more than manual throughput would; Advisor's cost recommendations say the same.

Metadata and transient 429s

Creating, reading or listing databases and containers, and reading offers, draws on a system-reserved budget that provisioned throughput doesn't touch. Insights, System, Metadata Requests By Status Code shows them. The fixes are in the client, not the account: one CosmosClient for the life of the process (initialising it fetches account metadata), container and database names from configuration rather than ReadContainerAsync on every request, and a backoff on any metadata call you must make. A transient service error 429 wants a retry and nothing else; if it persists for minutes, open a support ticket.

Retry in your own code

The SDK does the first nine. If you change the defaults, change both: in .NET, MaxRetryAttemptsOnRateLimitedRequests and MaxRetryWaitTimeOnRateLimitedRequests on CosmosClientOptions (or WithThrottlingRetryOptions on the builder); the minimum wait interval is a second. When the exception does reach you, read RetryAfter and honour it rather than hammering, and for 449 conflicts add a random element to the backoff, because many writers retrying on the same schedule collide again. Don't retry a 408 on a create: the service may have created the item, and the retry becomes a 409.

Sources

Did this fix it?

Comments

No comments yet. Share what worked for you.

Sign in to comment. Comments show under a display name you choose, never your email.