Hi there, this is another post (the sixth, I guess or something along those lines) in the One Tool, All APIs: PowerShell series. Previous posts:
- One Tool, All APIs: PowerShell As The All-Purpose API Client
- One Tool, All APIs: API Authentication With PowerShell
- One Tool, All APIs: API Parameters with PowerShell
- One Tool, All APIs: Construct API Headers with PowerShell
- One Tool, All APIs: Cookies & Session Control with PowerShell
This time, we will look into API limits and throttling and what we can do about it in PowerShell. As a tradition, we are starting with the basics.
In a nutshell, API throttling is used to control the rate at which clients can make requests to an API. Usually, it helps prevent abuse, maintain system stability, and manage server resources. Main goals are protecting API provider’s infrastructure and ensuring fair usage among consumers.
There is no specific RFC that outlines how throttling and limitations work but there are bits and pieces in few RFCs related to these topics.
RFC 7230 (HTTP/1.1 Message Syntax and Routing) – focuses on the syntax and routing of HTTP messages, defines the structure of HTTP messages, including requests and responses. As mentioned previously, it doesn’t explicitly address API throttling and limitations but provides the foundational standards for how HTTP messages (and responses) are structured and processed.
RFC 6585 (Additional HTTP Status Codes) – introduces additional HTTP status codes to address specific scenarios. The status code 429 Too Many Requests is relevant to throttling scenarios. It is a tool for indicating throttling, but the RFC doesn’t go into specific details on how throttling should be implemented; it just introduces the status code for use in such cases.
What can we do about it
Typically, all controls are in the hands of the API providers and implementors, but there are several strategies we can apply to handle or work around rate limits.
Progressive Retry Delay
This one is pretty self-explanatory, when you receive ‘429 Too many requests’ response, you simply wait for an increasing amount of time before trying again. This help to avoid immediate rejection.
Example
function Invoke-APIProgressiveDelay {
[CmdletBinding(SupportsShouldProcess=$true,
ConfirmImpact='Medium')]
param(
#API endpoint address
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[URI]$apiEndpoint,
#HTTP Method
[Microsoft.PowerShell.Commands.WebRequestMethod]$Method="GET",
# Maximum number of attempts
[int]$maxAttempts = 5,
# Initial delay in seconds
[int] $baseDelay = 1
)
for ($attempt = 1; $attempt -le $maxAttempts; $attempt++) {
try {
Write-Verbose "Trying to make API request to $apiEndpoint"
$response=Invoke-RestMethod -Uri $apiEndpoint -Method GET
Write-Verbose "Response received, break out of the loop and return the response"
$response
break
}
catch {
if ($_.Exception.Response.StatusCode -eq 429) {
$delay = [math]::Pow(2, $attempt - 1) * $baseDelay
Write-Verbose "429 Too Many Requests, applying exponential backoff: $delay second(s)"
Start-Sleep -Seconds $delay
}
else {
# Handle other exceptions
throw $_.Exception
}
}
}
}
For testing I will be using the httpstat.us service that is used for generating different HTTP codes.
Specifically, we will utilize range functionality so simulate occasional 429 response code.
1..10 | foreach -Parallel {function Invoke-APIProgressiveDelay {
[CmdletBinding(SupportsShouldProcess=$true,
ConfirmImpact='Medium')]
param(
#API endpoint address
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[URI]$apiEndpoint,
#HTTP Method
[Microsoft.PowerShell.Commands.WebRequestMethod]$Method="GET",
# Maximum number of attempts
[int]$maxAttempts = 5,
# Initial delay in seconds
[int] $baseDelay = 1
)
for ($attempt = 1; $attempt -le $maxAttempts; $attempt++) {
try {
Write-Verbose "Trying to make API request to $apiEndpoint"
$response=Invoke-RestMethod -Uri $apiEndpoint -Method GET
Write-Verbose "Response received, break out of the loop and return the response"
$response
break
}
catch {
if ($_.Exception.Response.StatusCode -eq 429) {
$delay = [math]::Pow(2, $attempt - 1) * $baseDelay
Write-Verbose "429 Too Many Requests, applying exponential backoff: $delay second(s)"
Start-Sleep -Seconds $delay
}
else {
# Handle other exceptions
throw $_.Exception
}
}
}
};Invoke-APIProgressiveDelay -apiEndpoint "https://httpstat.us/Random/200,429,429" -Verbose} -Verbose
Retry-After Header
The strategy revolves around checking the ‘Retry-After header in API response, as it tells how long (in seconds) client should wait before making another request. If you honor these values, you will avoid excessive requests.
Example
$apiEndpoint="httpstat.us/200"
do
{
Write-Output "Performing API request to $apiEndpoint"
$response = Invoke-WebRequest -Uri $apiEndpoint -Headers @{"X-HttpStatus-Response-Retry-After"=$(Get-Random -Min 1 -Max 25)}
$response | select-Object StatusCode,RawContent | fl
Write-Output "Waiting for $($response.Headers.'Retry-After') second(s) before next request"
Start-Sleep -Seconds $($response.Headers.'Retry-After')
}
while ($response.Headers.'Retry-After')
Rate Limit Monitoring
This one is a complementary strategy, basically you track API usage and stay within specified limits. You monitor response headers and stay within the limits imposed.
Typically, there are 3 headers that informs API consumers about the limits and the current usage:
X-Rate-Limit-LimitX-Rate-Limit-RemainingX-Rate-Limit-Reset
The tricky part is that there are no unified standards for the header values… Name of the headers are pretty self-explanatory but there are myriad of variations and different implementation of them.
X-Rate-Limit-Limit– can be the ultimate number of requests allowed for the client for this specific endpoint OR number of allowed requests per hour, minute etcX-Rate-Limit-Remaining– number of requests that left for the client BUT it is wild west regarding the rate windows it can be hours, minutes, months etc.X-Rate-Limit-Reset– time in UTC epoch seconds before rate limits reset or at which the current rate limit will be reset etc
The universal advice RTFM is very applicable, and I would say required here, as API developers have a lot of flexibility re how exactly they will implement the aforementioned headers.
$response = Invoke-WebRequest -Uri $apiEndpoint -Method GET -Headers $h
#Code to use response goes here.
$rateLimit = $response.Headers["X-Rate-Limit-Limit"]
$rateRemaining = $response.Headers["X-Rate-Limit-Remaining"]
$rateReset = $response.Headers["X-Rate-Limit-Reset"]
#Pause execution if we are close to hitting the rate limit
if ($rateRemaining -le 2) {
# Calculate time to wait until rate limit reset, adding a buffer of 1 second
$resetTime = (get-date).AddSeconds($rateReset).ToUniversalTime()
$timeToWait = $resetTime - $((Get-Date).ToUniversalTime())+ (New-TimeSpan -Seconds 1)
Write-Output "Approaching rate limit. Pausing for $timeToWait"
Start-Sleep -Seconds $timeToWait.TotalSeconds
}
Example
For our example, we assume the following values, the maximum number of requests per hour (X-Rate-Limit-Limit) – 3600, the number of requests remaining for the current hour (X-Rate-Limit-Remaining) – 8 and the last but not least is the rate windows second remaining for rate limit reset (X-Rate-Limit-Reset) – variable depends at what time you are consuming the API endpoint.
$h=@{
#Max number of requests allowed in an hour (1 request per second)
"X-HttpStatus-Response-X-Rate-Limit-Limit"=3600
#Number of remaining requests
"X-HttpStatus-Response-X-Rate-Limit-Remaining"=8
#Seconds left to the next hour, when rate limit should reset
"X-HttpStatus-Response-X-Rate-Limit-Reset"=$((Get-Date $([datetime](Get-Date).AddHours(1).ToString("yyyy-MM-dd HH:00:00")).ToUniversalTime() -UFormat %s) - [math]::Truncate((Get-Date).ToUniversalTime().Subtract([datetime]"1970-01-01").TotalSeconds))
}
$apiEndpoint="httpstat.us/200"
$try=1
do
{
try {
$h.'X-HttpStatus-Response-X-Rate-Limit-Remaining'--
Write-Output "($try) Performing API request to $apiEndpoint"
$response = Invoke-WebRequest -Uri $apiEndpoint -Method GET -Headers $h
Write-Output "Response received. Proceeding with response parsing and consuming logic"
$rateLimit = $response.Headers["X-Rate-Limit-Limit"]
$rateRemaining = $response.Headers["X-Rate-Limit-Remaining"]
$rateReset = $response.Headers["X-Rate-Limit-Reset"]
Write-Output "Requests remaining: $rateRemaining"
#Pause execution if we are close to hitting the rate limit
if ($rateRemaining[0] -le 2) {
# Calculate time to wait until rate limit reset, adding a buffer of 1 second
$resetTime = (get-date).AddSeconds($rateReset[0]).ToUniversalTime()
$timeToWait = $resetTime - $((Get-Date).ToUniversalTime())+ (New-TimeSpan -Seconds 1)
Write-Output "Approaching rate limit. Pausing for $timeToWait"
Start-Sleep -Seconds $timeToWait.TotalSeconds
}
$try++
}
catch {$response = $_.Exception.Response}
}
until ($try -eq 10)
Adjust Request Frequency
The crux of this strategy is in spreading your requests evenly over time and not submitting them in bursts. Idea is to stay within the rate limits and reduce chances of hitting throttling thresholds.
Example
$apiEndpoint="httpstat.us/200"
#Number of requests you plan to make
$requests = 151
# Time window in seconds
$timeFrame = 3600
# Calculate the delay to spread them evenly over the time window
$delay = $timeFrame / $requests
for ($i = 1; $i -le $requests; $i++) {
Write-Output "Invoking API endpoint $apiEndpoint"
$response=Invoke-RestMethod -Uri $apiEndpoint
$response
Write-Output "Waiting for $delay second(s) before next request"
Start-Sleep -Seconds $delay
}
Optimize Batch Requests
This one is pretty obvious if API allows/supports batch requests consider groping multiple requests together. It will be more efficient and will reduce the number of API calls. Read documentation carefully and try to group/transform your requests using batch method.
User-Agent Rotation
The approach here is a bit sleazy and I would say lays the gray area. Sometimes, rate limit is applied by checking the user agent. By frequently changing user agent in the request headers might help to distribute a load.
Example
We will use The Latest and Most Common User Agents List (Updated Weekly) service to get the file with the most common User Agents.
$userAgentTSV=@" Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.3 34.12 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.3 14.12 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124. 13.53 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0. 12.35 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.3 4.71 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 Edg/117.0.2045.4 4.12 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 OPR/108.0.0. 2.35 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0. 2.35 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 Edg/112.0.1722.5 1.76 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.5 1.18 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.3 1.18 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.3 1.18 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 OPR/108.0.0. 0.59 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123. 0.59 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.3 0.59 Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115. 0.59 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.14 0.59 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.10 0.59 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/118. 0.59 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.3 0.59 Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Geck 0.59 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.2 0.59 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 Edg/112.0.1722.3 0.59 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0. 0.59 "@
Now we will randomly get the user agent from the list and use it during our API request.
$apiEndpoint="httpstat.us/200"
1..10 | %{
Write-Output "Try #$_"
#Getting random user agent from the list
$useragent=($($userAgentTSV -split "`n" | Get-Random) -split "`t")[0]
Write-Output "Invoking API endpoint $apiEndpoint"
Write-Output "Using User-Agent: $useragent"
$response = Invoke-RestMethod -Uri $apiEndpoint -Headers @{"User-Agent" =$userAgent}
$response
}
Conclusion
At the end of the day, each of the strategies and/or approaches will not provide you the best single way to workaround or keep your requests under limit. Thes first step to read the API documentation and clearly understand the limit and thresholds. The key is to mix different strategies and use layered approach, so if one of the strategies will not yield any results you will still have a cushion of other ones to help you out.
Icons created by Freepik – Flaticon.
Thanks a lot for reading.
