WebGrabber Maximum Connections and Conversion Timeouts

WebGrabber Maximum Connections and Conversion Timeouts

When using WebGrabber 2013 and above under heavy use in certain environments and converting some URLs, WebGrabber may appear to time out or fail, or you may be advised of a threat/intrusion detected.

Heavy use can be described as WebGrabber set for multiple threads at hundreds of conversions each. Example: 6 threads with 800 conversions each.

These timeouts and/or failures are not a limitation of WebGrabber. There is, instead, a limitation or restriction between WebGrabber and the URL you are attempting to convert.

These restrictions can be placed into two categories:

• Allocated connections reached
• Connections limited by a website

Allocated Connections Reached

Firewalls, routers, network appliances, DNS gateways and/or Internet blocking/filtering software are often used to provide security against intrusion or to limit the number of connections that can be made to an outside URL. The multiple connection attempts made during a short span of time may cause you to hit the maximum number of connections currently alotted, resulting in a timeout or error.

Some error messages include:

• "Network intrusion detected."
• "Packet dropped by firewall."
• "Maximum connections reached."

You may be able to resolve the issue by contacting your IT department or System Administrator for advice. Your IT department or System Administator may review and adjust your corporate network policy's rules and restrictions where WebGrabber is concerned. Otherwise, you may need to run fewer threads and/or conversions.

Connections Limited by a Website

Many websites, such as google.com, limit the number of connections made to the site over a specific time period (i.e. connection attempts faster than are humanly possible). This is done as a precautionary security measure to avert malicious attempts at intrusion or denial of service to the site. Although your conversion attempts with WebGrabber are not malicious, they may be perceived as an attack, and your connection may be refused.

Some symptoms and errors include:

• Redirected to another site or to a page with language similar to: "Too many requests".
• "Connection refused."
• Conversion fails with or without an error, but you can access the page manually.

As there is nothing you can do to change the policies of other websites, ActivePDF recommends running fewer WebGrabber threads and/or conversions.