Micrograbs

[Max]

Shepherd should detect "microgaps" in data -- holes that aren't big enough to trigger a policy violation and thus a re-fetching of the entire day -- and send out grabbers to fetch just the necessary data. Rex and the abc grabbers could do this.

Logic:

(1) If we have a hole that fails the policy test (e.g. is in prime time, or of significant size), always send another grabber out to fill it if possible. If the hole is small, then prefer grabbers that support micro-grabbing, but either way, send out something.

Unless data for the entire day(s) is required, send the command-line option "--gaps_file <s>", where file <s> contains an eval-uable data structure called $gaps. The normal command-line parameters (such as --days and --channels_file) are also sent. Grabbers that do not support micro-grabbing are expected to ignore the --gaps_file option.

(2) If we have a hole that does NOT fail the policy test, but there is a grabber available that supports micro-grabbing data for that region/day/channel (as per its .conf file), then send it out with the same options as above. If there is no valid micro-grabber, live with the hole rather than sending out a non-micro-grabber.

[Max]

Done!

yahoo7widget could now ditch its hybrid gap-filling routine.