Configure Crawler Impact Rules

At the SharePoint farm level, the crawler impact rules can be used to modify the effect that search has on the sites it is crawling.

Crawler impact rules are policies that allow administrators to constrain the volume of requests that SharePoint makes to a particular site or server. The purpose of these rules is to throttle the number of requests for pages and documents that SharePoint makes to a server to match the rate at which the server can respond. For example, if the SharePoint server is more powerful than the target server that it is crawling, it might request pages faster than the target can respond. This can choke the target and cause unanswered requests to queue up on the SharePoint server, reducing overall performance. By reducing the number of simultaneous requests that the SharePoint server makes, you can improve the performance of both systems.

To add a crawler impact rule, do the following:

1. From SharePoint 2010 Central Administration, click General Application Settings.

2. Under Search, click Crawler Impact Rules.

3. Click Add Rule.

4. In the Site text box, shown in Figure 8.6, enter the name of the site or server that is being crawled without the protocol prefix. For example, enter http://accounting.synergy.com.

Figure 8.6: Add Crawler Impact Rule

5. Enter the number of simultaneous requests that SharePoint will make to the server. It is generally not necessary to increase the number above the default of 8 unless the target that is being crawled can respond quickly enough to answer all the requests at once. In that case, a higher setting could reduce the crawl duration. In most cases, a crawl rule is used to reduce the number of requests even to the point of only one at a time if that is all the target server can handle.

6. Click OK.