Ticket #19 (closed defect: fixed)
Grabbers should maybe timeout faster
|Reported by:||max||Owned by:||lincoln|
By chance my network went down for 2 hours today, and it was interesting to see what effect this had on Shepherd.
- Shepherd started running at 11.15am
- By 11.30am, grabber #1 (yahoo7widget) had timed out and grabber #2 (rex) was halfway through timing out. Rex took 21 minutes to time out altogether.
- By 11.45am, grabber #3 (jrobbo) had timed out (quickly, I think) and grabber #4 (abc2_website) had just started.
- At 12 noon, abc2_website was still futilely trying to download summary pages.
- At 12.15pm, abc2_website was still running.
- At 12.30pm, abc2_website had timed out and grabber #5 (abc_website) was running.
- The network was then restored. abc_website completed and grabber #6 (ninemsn) started.
When I noticed that a Tor-less ninemsn was running, I killed it rather than have it take 12 hours or whatever it is.
Not sure what the ideal behavior is here, but if grabbers timed out more sensibly, it'd be less likely that we'd hit situations where the network is down for earlier grabbers but back up for later ones.
If a grabber can't download anything it should really time out within a few minutes, I'd think.