A web scraper that performs well across a few thousand pages can behave very differently when the workload grows to hundreds of thousands or millions of URLs.
At smaller volumes, performance improvements can seem straightforward: run more requests in parallel, add more capacity, or increase available infrastructure.
Eventually, however, simply increasing the number of requests stops producing proportional improvements.
Target websites may begin responding more slowly. Rate limits appear. Connections time out. Retry traffic increases. Processing queues grow faster than they can be consumed. Proxy performance can vary between routes.
At this point, scaling becomes less about generating more traffic and more about controlling traffic effectively.
Three concepts become particularly important:
Concurrency — how many operations happen at the same time.
Rate limiting — how quickly requests are sent.
Backpressure — how the system responds when one part of the pipeline cannot keep up.
Understanding how these mechanisms work together is an important part of operating reliable web scraping workloads at scale.
Why More Concurrency Doesn't Always Mean More Throughput
Concurrency allows a scraper to perform multiple network operations simultaneously.
Because much of the time spent scraping is waiting for remote servers to respond, concurrency can significantly improve performance.
A scraper processing requests sequentially might spend much of its runtime waiting on network responses. Allowing multiple requests to remain in flight keeps the system productive during those waiting periods.
The problem appears when concurrency is treated as something that should simply be increased indefinitely.
Imagine increasing a scraper from 20 concurrent requests to 100.
Throughput might improve significantly.
Increasing from 100 to 500 may still provide an improvement.
Increasing from 500 to 5,000, however, might produce very different results.
Instead of increasing useful throughput, the scraper may begin experiencing:
higher latency,
more connection failures,
increased timeouts,
greater memory consumption,
destination throttling,
overloaded proxy connections,
and more retries.
There is therefore usually a point where additional concurrency produces diminishing returns.
The objective isn't to achieve the highest possible concurrency.
It is to find a level of concurrency that produces the highest sustainable throughput.
Concurrency Should Reflect the Destination
Not every website behaves the same way.
A large global platform may comfortably handle considerably more traffic than a small specialist website running on modest infrastructure.
Response times also vary significantly.
One destination might consistently respond within 200 milliseconds while another regularly takes several seconds.
Applying the same concurrency configuration everywhere can therefore produce poor results.
Large scraping workloads commonly benefit from controlling concurrency at more than just the overall crawler level.
For example, a crawler might allow substantial overall concurrency while restricting the number of simultaneous requests sent to individual domains.
This provides two benefits.
First, a slow destination does not consume an excessive portion of the scraper's available capacity.
Second, traffic can be distributed more appropriately across different websites.
This becomes particularly important when one scraping workload contains URLs across hundreds or thousands of domains.
Concurrency and Rate Limits Solve Different Problems
Concurrency and request rate are sometimes treated as interchangeable concepts.
They aren't.
Concurrency controls how many requests are currently in progress.
Rate limiting controls how frequently new requests are sent.
Consider a website that responds very quickly.
Even relatively modest concurrency can generate a surprisingly high number of requests per second.
A scraper running 20 concurrent connections against a fast destination might generate far more traffic than expected.
Conversely, a slow website could have many concurrent connections while still receiving relatively few requests per second.
For this reason, mature scraping systems often control both.
Concurrency protects the scraper and its supporting infrastructure from having too much work in flight.
Rate limiting controls how quickly traffic reaches individual destinations.
Together, they provide much better control over scraping behaviour.
Rate Limiting Should Usually Be Destination-Aware
A single request limit across an entire scraping operation is rarely ideal.
Suppose a crawler is collecting information from 50 unrelated websites.
One website might comfortably handle a certain request rate while another begins slowing down or returning rate-limit responses at a much lower level.
Reducing the entire crawler to accommodate the slowest destination wastes available capacity.
Instead, large-scale scraping systems commonly maintain limits for individual destinations or groups of destinations.
This allows the crawler to continue processing healthy destinations while slowing traffic to those showing signs of pressure.
Destination-aware controls are also useful when dealing with explicit rate limits.
HTTP 429 Too Many Requests, for example, is a strong indication that traffic should slow down.
Rather than treating that response as simply another failed request, the crawler can use it as feedback.
That leads to an important principle of large-scale scraping:
Traffic controls should respond to what the destination is telling you.
Static Limits Are Useful, but Conditions Change
A configuration that works today may not work tomorrow.
Website infrastructure changes.
Traffic conditions change.
Network routes change.
Response times fluctuate.
A destination that normally responds quickly might temporarily become significantly slower.
This means fixed rules such as:
Always send 20 requests per second.
can eventually become inefficient.
More sophisticated scraping operations use observed behaviour to adjust traffic.
If responses remain fast and successful, the system may have room to process more work.
If latency begins increasing substantially, concurrency can be reduced.
If rate-limit responses increase, request frequency can be lowered.
If error rates suddenly rise, the system can temporarily become more conservative.
This does not require extremely complex automation.
Even simple feedback-based adjustments can make a scraper considerably more resilient than relying entirely on fixed limits.
Backpressure: The Scaling Problem That Is Easy to Miss
Concurrency and rate limiting mostly concern how quickly requests are made.
But fetching pages is only one part of a scraping workload.
A typical data collection process may involve several stages:
Fetch → Parse → Transform → Validate → Store
Each stage operates at a different speed.
Suppose pages are being downloaded faster than they can be parsed.
The downloaded pages begin accumulating.
Or perhaps parsing is fast, but writing extracted records to a database becomes slow.
Now records begin waiting for storage.
If the difference is temporary, this may not matter.
If it continues for hours, queues can become enormous.
Eventually the scraper may consume excessive memory, run out of storage, or create increasingly long processing delays.
This is where backpressure becomes important.
What Backpressure Actually Means
Backpressure simply means allowing slower parts of a system to influence faster parts.
Consider a supermarket checkout.
If customers enter the store faster than checkout counters can process them, queues grow.
At some point, continuously allowing more people through the doors makes the situation worse.
A scraping pipeline faces the same problem.
If the database can process 5,000 records per minute while the scraper generates 10,000 records per minute, the difference has to accumulate somewhere.
Backpressure allows the system to recognise that imbalance.
The fetching stage might temporarily slow down.
Workers might stop accepting additional tasks.
New jobs might remain in a queue until capacity becomes available.
The important principle is that the system does not continue producing work indefinitely when downstream components cannot consume it.
Queues Help Absorb Short-Term Differences
Queues are commonly used between stages of larger scraping systems.
They allow one part of the system to continue operating when another temporarily slows down.
For example, if a database experiences a short performance problem, extracted records can wait in a queue rather than causing the entire scraping operation to fail.
Queues are therefore extremely useful.
But they can also hide problems.
A queue that grows continuously isn't solving a capacity problem. It is postponing it.
If 10,000 items enter a queue every minute while only 8,000 leave, the backlog grows by 2,000 items every minute.
Eventually something must change.
This is why queue growth and age can be more informative than queue size alone.
A large queue being processed quickly may be healthy.
A smaller queue whose oldest item has been waiting for hours may indicate a serious bottleneck.
Retries Can Make Overload Worse
Retries are essential in web scraping.
Networks fail.
Connections time out.
Servers occasionally return temporary errors.
Proxy routes can become unavailable.
A second attempt often succeeds.
But retries become dangerous when failures happen at scale.
Imagine a scraper sending 10,000 requests and suddenly experiencing a high failure rate.
If every failure is immediately retried, the scraper generates even more traffic at exactly the moment when something is already going wrong.
Those retries may also fail and generate further retries.
This can create a retry storm.
Instead of helping the system recover, retries increase the load.
Good Retry Behaviour Includes Delay
One of the simplest protections is to introduce increasing delays between retry attempts.
Instead of:
Failure → retry immediately → retry immediately again
the crawler waits progressively longer between attempts.
Random variation can also be added to those delays.
This prevents thousands of workers that encountered failures simultaneously from retrying simultaneously.
More importantly, not every failure should be retried.
A temporary network timeout may justify another attempt.
A 404 Not Found generally doesn't.
A 429 Too Many Requests is usually a reason to reduce traffic rather than repeatedly resend the same request.
Repeated server errors may indicate that waiting is preferable to retrying aggressively.
Understanding why a request failed is therefore just as important as deciding whether it should be retried.
Proxy Rotation Doesn't Replace Traffic Control
Proxies play an important role in many large-scale web data collection workloads.
Residential and datacenter proxy networks can provide access to different network routes, locations, and IP addresses.
But increasing the size of a proxy pool does not remove the need for rate limiting.
Imagine distributing requests across thousands of IP addresses while sending them all to the same website.
From the scraper's perspective, traffic has been distributed across the proxy network.
From the destination's perspective, however, it may still be receiving an excessive volume of requests.
Proxy rotation and traffic management therefore solve different problems.
Proxy infrastructure determines how requests reach a destination.
Rate limiting determines how much traffic reaches that destination.
A well-designed scraping operation needs both.
Proxy Performance Also Affects Concurrency
At larger volumes, proxy performance itself becomes another variable.
Not every route will have identical latency or reliability.
Residential proxies, for example, can naturally have more variable network characteristics than highly predictable datacenter connections.
That matters when determining concurrency.
If average response latency increases, more requests remain active for longer.
That means the same request rate can produce significantly more concurrent connections.
Scraping teams should therefore monitor proxy performance alongside destination performance.
Useful indicators include:
connection success rate,
request latency,
timeout rate,
geographic availability,
session stability,
and destination success rate.
Services such as Raspbytes residential and datacenter proxies can provide the proxy layer for these workloads, while concurrency, rate limiting, retry behaviour, and destination policies remain part of the scraping application's own traffic-management strategy.
The distinction is important: proxies provide network capacity and routing options, while the crawler still needs to use that capacity intelligently.
Don't Measure Scale Only in Requests per Second
Requests per second is one of the easiest scraping metrics to measure.
It can also be misleading.
Consider two scraping runs.
The first generates 5,000 requests per second but succeeds on only 70% of them.
The second generates 4,000 requests per second with a 95% success rate.
The first produces approximately 3,500 successful responses per second.
The second produces approximately 3,800.
The apparently slower scraper is actually collecting more useful data.
It is also generating fewer failed requests and potentially consuming less proxy bandwidth and infrastructure.
This is why large scraping operations increasingly need to think in terms of useful throughput rather than raw traffic volume.
Useful metrics include:
successful responses per second,
success rate,
average and percentile latency,
retry rate,
rate-limit responses,
queue growth,
processing delay,
proxy success rate,
and cost per successful result.
These provide a much clearer picture of whether increasing scale is actually improving performance.
Managing Different Workloads at Scale
Large scraping operations rarely consist of one continuous workload.
There may be scheduled crawls, incremental updates, newly discovered URLs, retries, and time-sensitive collections running at the same time. These workloads can have very different requirements.
A daily catalogue refresh, for example, may tolerate being processed gradually over several hours. A collection tracking rapidly changing information may need to complete much sooner.
Treating every request with the same priority can therefore lead to inefficient use of capacity.
At scale, it is useful to think about how available capacity should be distributed across workloads. Time-sensitive collection may be prioritised, while less urgent jobs can use spare capacity or run at a lower rate.
This becomes particularly important when the system is already under pressure. Rather than allowing every workload to compete equally for workers, connections, proxy capacity, and downstream processing, lower-priority work can be slowed while more important collection continues.
Prioritisation should still operate within the broader traffic controls discussed earlier. A high-priority workload should not bypass destination rate limits simply because it needs to finish sooner.
The goal is not necessarily to make every scraping job complete as quickly as possible. It is to ensure that available capacity is used according to the requirements of the workload without overwhelming the rest of the pipeline.
Find the Bottleneck Before Increasing Capacity
When scraping throughput starts falling behind demand, the natural response is often to increase capacity.
But more capacity only helps when the constraint is within the part of the system being expanded.
A growing backlog could have many causes. The destination may be responding more slowly or enforcing rate limits. Proxy routes may be experiencing higher latency or connection failures. Parsing and transformation may be taking longer than expected. Storage systems may be struggling to keep pace with incoming data.
Increasing request concurrency in any of these situations can make the problem worse rather than improve throughput. More requests create additional pressure while the underlying constraint remains unchanged.
This is why scaling decisions should start with identifying where throughput is actually being limited.
Useful signals include changes in response latency, success rates, rate-limit responses, retry volume, queue growth, proxy performance, processing time, and storage latency. Looking at these together helps distinguish between a temporary slowdown and a genuine capacity constraint.
The principle is simple: scale the bottleneck, not just the volume of activity around it.
Effective scaling is not about continuously adding resources. It is about understanding which part of the collection process is limiting useful throughput and addressing that constraint without creating pressure elsewhere.
The Goal Is Sustainable Throughput
The biggest misconception about scaling web scraping is that the objective is to send as many requests as possible.
It isn't.
The objective is to collect the required data reliably, efficiently, and at a rate that the entire system can sustain.
Concurrency provides parallelism.
Rate limiting controls traffic.
Backpressure prevents downstream bottlenecks from becoming system-wide failures.
Queues absorb temporary differences in processing speed.
Retry policies help recover from transient failures without creating additional overload.
Proxy networks provide the routing capacity required for many distributed collection workloads.
None of these mechanisms work particularly well in isolation.
At small scale, their importance may not be obvious because there is enough spare capacity to hide inefficiencies.
At large scale, those inefficiencies become visible very quickly.
The difference between a scraper capable of generating enormous amounts of traffic and a scraper capable of operating reliably at scale is ultimately control.
The most effective systems don't simply know how to speed up.
They also know when to slow down.
A Reliable Proxy Layer for Your Scraping Workloads
As scraping workloads grow, proxy infrastructure often becomes an important part of the networking layer.
Raspbytes provides residential and datacenter proxies for web scraping, data collection, automation, and other proxy-enabled workloads.
Use Raspbytes alongside your existing scraping stack to add proxy routing and IP diversity while retaining control over your own concurrency, rate limits, sessions, and collection strategy.
Explore Raspbytes Proxies and start building more reliable web data collection workflows.