Grabbers should maybe timeout faster

[max]

By chance my network went down for 2 hours today, and it was interesting to see what effect this had on Shepherd.

1. Shepherd started running at 11.15am
2. By 11.30am, grabber #1 (yahoo7widget) had timed out and grabber #2 (rex) was halfway through timing out. Rex took 21 minutes to time out altogether.
3. By 11.45am, grabber #3 (jrobbo) had timed out (quickly, I think) and grabber #4 (abc2_website) had just started.
4. At 12 noon, abc2_website was still futilely trying to download summary pages.
5. At 12.15pm, abc2_website was still running.
6. At 12.30pm, abc2_website had timed out and grabber #5 (abc_website) was running.
7. The network was then restored. abc_website completed and grabber #6 (ninemsn) started.

When I noticed that a Tor-less ninemsn was running, I killed it rather than have it take 12 hours or whatever it is.

Not sure what the ideal behavior is here, but if grabbers timed out more sensibly, it'd be less likely that we'd hit situations where the network is down for earlier grabbers but back up for later ones.

If a grabber can't download anything it should really time out within a few minutes, I'd think.

[max]

Fixed in Rex by [340].

[max]

I ran a test to see where we still have issues. I found these grabbers timed out nice and quickly when there's no network connectivity:

1. oztivo (1 second)
2. yahoo7widget (30 seconds)
3. jrobbo (30 seconds)
4. ninemsn with Tor (50 seconds)
5. rex (1 minute)
6. sbsnews_website (1.5 minutes)
7. ten_website (5 minutes)

But these grabbers take much longer than they should:

1. yahoo7web (2 hours 15 minutes)
2. ninemsn without Tor (50 minutes)
3. abc_website (35 minutes)
4. abc2_website (35 minutes)

Assigning to Lincoln because these are his grabbers. :)

[lincoln]

fixed in abc_website/abc2_website in [521]. fixed in ninemsn in [522]. fixed in yahoo7web in [523].

All should now timeout in under 5 minutes with somewhere between 3-5 retries before failing.