Opened 17 years ago

Closed 17 years ago

Last modified 13 years ago

#24 closed enhancement (fixed)

Micrograbs

Reported by: Max Owned by:
Priority: minor Milestone: 1.0
Component: Shepherd Version: 0.4
Keywords: Cc:

Description

Shepherd should detect "microgaps" in data -- holes that aren't big enough to trigger a policy violation and thus a re-fetching of the entire day -- and send out grabbers to fetch just the necessary data. Rex and the abc grabbers could do this.

Logic:

(1) If we have a hole that fails the policy test (e.g. is in prime time, or of significant size), always send another grabber out to fill it if possible. If the hole is small, then prefer grabbers that support micro-grabbing, but either way, send out something.

Unless data for the entire day(s) is required, send the command-line option "--gaps_file <s>", where file <s> contains an eval-uable data structure called $gaps. The normal command-line parameters (such as --days and --channels_file) are also sent. Grabbers that do not support micro-grabbing are expected to ignore the --gaps_file option.

(2) If we have a hole that does NOT fail the policy test, but there is a grabber available that supports micro-grabbing data for that region/day/channel (as per its .conf file), then send it out with the same options as above. If there is no valid micro-grabber, live with the hole rather than sending out a non-micro-grabber.

Change History (2)

comment:1 Changed 17 years ago by Max

Resolution: fixed
Status: newclosed

Done!

yahoo7widget could now ditch its hybrid gap-filling routine.

comment:2 Changed 13 years ago by krtbwy

Keywords: I05LDG added
Milestone: 1.0
Note: See TracTickets for help on using tickets.