Context Navigation

Changes between Version 71 and Version 72 of FAQ

Timestamp:: Sep 30, 2013, 6:40:06 PM (10 years ago)
Author:: Max
Comment:: update!

Legend:

: Unmodified
: Added
: Removed
: Modified

FAQ

-                      v71
+                      v72
 === What is Shepherd? ===
 Shepherd is an attempt to reconcile many different tv_grab_au scripts and make one cohesive reliable data set.
+Shepherd is software for compiling an Electronic Program Guide (EPG), by managing a flock of guide data grabbers and postprocessors.
 === How does it work? ===
 …
 Shepherd is designed to be ''future proof'', never requiring manual intervention once initially installed and configured.  Shepherd will automatically update itself with fixes, enhancements and additional ''plugin'' components as and when they become available.
+By default, Shepherd performs one full run per day, gathering guide data for the next 8 days. It also autorefreshes the current day's data once every four hours or so, to catch for last-minute schedule changes.
 The [wiki:shepherd_logic] wiki page contains a more complete technical description of the various stages of Shepherd and how it works.
 …
 In theory, Shepherd (and its underlying components) will run on any operating system that supports Perl, as all scripts are currently written in Perl.
+In practice, the developers all use Linux and MythTV, and that is what is known to work.
+No effort has been put into making Shepherd work under Microsoft Windows or Windows Media Centre Edition, although it really shouldn't be too hard to get that working if anyone was motivated enough to do so.
+In practice, the developers all use Linux, so that's what is tested. There are Shepherd users running on [wiki:ShepherdForWindows Windows] and [wiki:Installation#AppleOSX10.8 OS X] but these require more fiddling.
+=== Does Shepherd only work with MythTV? ===
+Shepherd will work with any PVR-type software capable of reading XMLTV (which is most of them). MythTV users get a relatively painless installation, since Shepherd can set itself up automatically, but MythTV is not a pre-requisite. Shepherd is often used on alternatives, such as Freevo.
+If you are having trouble getting Shepherd to work outside the usual environment of Linux + MythTV, don't hesitate to write in to our mailing list for help.
 === Is Shepherd legal? ===
 Some of the grabbers used by Shepherd read web sites that say they don't want their data used in PVRs, but that [http://www.cse.unsw.edu.au/~willu/xmltv/index.html#Legal doesn't mean it's illegal]. Shepherd doesn't copy or distribute data, but rather allows individuals at home to read it via their PVRs. It operates in the same manner as a browser, sending HTTP requests and formatting the resultant HTML for display in a manner appropriate to the user.
+Some of the grabbers used by Shepherd read web sites that say they don't want their data used in PVRs, but that [http://www.cse.unsw.edu.au/~willu/xmltv/index.html#Legal doesn't mean it's illegal]. Shepherd doesn't host or distribute data, but rather allows individuals at home to access it via their PVRs. Like a browser, it sends HTTP requests to public websites and formats the result for display in a manner appropriate to the user.
 == Installing Shepherd ==
 …
 A summary of Shepherd's performance can be viewed by:
 {{{
 ~/.shepherd/tv_grab_au --status
+tv_grab_au --status
 }}}
 In particular, note the last line, which tells you the percentage of wanted data it acquired. If it's less than 100.00%, Shepherd wasn't able to completely data for all your channels over the next 7 days. If you haven't enabled all of Shepherd's grabbers, you will probably benefit from doing this.
 …
+=== How do I make Shepherd use a particular grabber? ===
+Generally it's best to let Shepherd decide which grabbers to use, and in which order. One of its main benefits is its fault-tolerance: guide data sources tend to be fragile and can stop working unexpectedly, but Shepherd will work regardless. Relying on a particular grabber always being there is, unfortunately, not safe.
+If you have a general preference for speed over quality, or vice versa, you can control this via [wiki:FAQ#CanImakeShepherdfaster the "--mode" option].
+To specify an exact order for grabbers, use the "--grabwith <grabber/s>" option. Shepherd will run the specified grabber(s) first, then others as needed to fill remaining holes in the data. For example:
+{{{
+~/.shepherd/tv_grab_au --grabwith oztivo,sbsnews_website
+}}}
+=== What is autorefresh? ===
+When it's been more than 4 hours but less than 22 hours since the last run, Shepherd will switch into autorefresh mode, gathering data for the current day only. It will output the result via STDOUT, as usual, and create the file ~/.shepherd/refresh.xmltv. This allows Shepherd to catch last-minute schedule changes. It tends to complete very quickly (for Shepherd), since even the slower grabbers are usually able to quickly verify if there have been any time or title changes since they last ran.
+If you don't wish Shepherd to enter autorefresh mode, send the --noautorefresh option.
+Shepherd will '''not''' enter autorefresh mode if you specify a number of --days, since this would require overriding your preference. But if you are happy for Shepherd to override it during autorefreshes only, send --allowautorefresh.
+Autorefresh does not touch the default output.xmltv file, so functions such as '''--refill-mythtv''' will use your last full successful run, not any subsequent autorefresh. You may wish to override this if you want to be able to rely on output.xmltv always containing the latest data, regardless of whether it's a single day's data or a week's. If so, use '''--output'''. For example, to always output into output.xmltv, even autorefreshes:
+{{{
+tv_grab_au --output ~/.shepherd/output.xmltv
+}}}
+In that case, you would probably want to set it as a default option, i.e.:
+{{{
+tv_grab_au --component-set shepherd:output=output.xmltv
+}}}
+Or to refill MythTV will the latest autorefresh data, rather than the latest full run:
+{{{
+tv_grab_au --refill-mythtv --output ~/.shepherd/refresh.xmltv
+}}}
 === Why is Shepherd so slow? ===
 …
 Since Shepherd runs in the background, how long it takes makes no difference. If you'd like it to complete faster, though, you may use [wiki:FAQ#CanImakeShepherdfaster the "--mode" option].
 === Can I make Shepherd faster?  ===
 By default Shepherd operates in "Quality" mode, whereby it selects grabbers based on maximizing data quantity and quality. This mode can take a long time: around 15-30 minutes under normal operating conditions, and several hours if it's being run for the first time, with no cache and needing to fill a full week.
 …
 To run Shepherd once in a particular mode:
 {{{
 ~/.shepherd/shepherd --mode speed
+tv_grab_au --mode speed
 }}}
 or to permanently set Shepherd in a particular mode:
 {{{
 ~/.shepherd/shepherd --component-set shepherd:mode=speed
+tv_grab_au --component-set shepherd:mode=speed
 }}}
 …
 If you want Shepherd to always be called as if it was sent a particular command-line option, you can use:
 {{{
 ~/.shepherd/tv_grab_au --component-set shepherd:<option>
+tv_grab_au --component-set shepherd:<option>
 }}}
 For example, this would make Shepherd always run as if called with the option "--grabwith=abc_website":
 {{{
 ~/.shepherd/tv_grab_au --component-set shepherd:grabwith=abc_website
+tv_grab_au --component-set shepherd:grabwith=abc_website
 }}}
 If you want to add multiple options they all need to be set with one command otherwise the final command will override any previously set commands. For example to add both "--notquiet" and "--grabwith=abc_website" :
 {{{
 ~/.shepherd/tv_grab_au --component-set shepherd:notquiet:grabwith=abc_website
+tv_grab_au --component-set shepherd:notquiet:grabwith=abc_website
 }}}
 You can also set default options for any component, e.g.:
 {{{
 ~/.shepherd/tv_grab_au --component-set abc_website:do-extra-days
+tv_grab_au --component-set abc_website:do-extra-days
 }}}
 To clear all default options, call --component-set with no argument. E.g.:
 {{{
+~/.shepherd/tv_grab_au --component-set shepherd
+}}}
+tv_grab_au --component-set shepherd
+}}}
 === Can I grab more than 7 days of data? ===
 (Note: the Shepherd default is now eight days.)
+(Note: The Shepherd default is now eight days.)
 You can ask Shepherd to try to grab however many days you like. Some channels in some regions offer up to 28 days of data; others as few as three or four. Generally, you can get at least 7 days for all but the community channels, and 14+ days for ABC and SBS.
 …
 }}}
+'''Note:''' This will disable autorefresh, which is where Shepherd refreshes the current day's data once every 4 hours, unless you also specify --allowautorefresh.
 === Can I get guide data from multiple different regions? ===
 …
 Users grabbing data from regions in different timezones should also ensure that each Shepherd instance has [wiki:FAQ#WhichtimezoneshouldIsetMythTVtoAutoNone1000 the correct TZ variable set].
 === Can I specify different configuration files for Shepherd to use? ===
 …
 Multiple versions of Shepherd will produce multiple output.xmltv files, which you will need to pass to your PVR. In MythTV, for example, you should not use the default cron job (which invokes mythfilldatabase rather than Shepherd), but rather create your own cron jobs to run each instance of Shepherd, then feed their output to MythTV [wiki:Installation#Multiplesources as described here].
-=== My high definition (HD) channels are missing programs? ===
-You must configure standard definition (SD) channels for the corresponding HD channels for them to be populated correctly.
-It is now possible to obtain fully populated HD channels with KNOWN HD programs flagged as HD.  This is the DEFAULT for new installs.  To enable on old installs, go to your shepherd install and execute:
-{{{
-rm ~/.shepherd/postprocessors/flag_aus_hdtv/flag_aus_hdtv.config
-}}}
-Then see [http://svn.whuffy.com/wiki/FAQ#HowdoIusesthenewhighdefinitionHDchannels How do I use the new high definition (HD) channels?]
-Originally the HD channels were only populated with KNOWN HD programs.  The idea is to have both SD and HD, and increase the priority of HD channels, so programs record as HD when available.  To change to this behaviour use these commands:
-{{{
-cd ~/.shepherd/postprocessors/flag_aus_hdtv
-ln -s ../../references/Shepherd
-./flag_aus_hdtv --set=action:copy
-rm Shepherd
-}}}
-WARNING: Some stations do upscaling from SD to HD; you could record the HD version but the SD version is at least half the size for the same detail.  Also some stations run a program of scenery in a loop; if you recorded their HD channel you would miss your program.
-Another way to obtain fully populated HD channels, is use the SD xmlids for the HD channels and remove your HD channels from shepherd BUT you will miss any SD channel and HD channel divergence.
-If any additional programs should be flagged HD, please let us known on our mail list.
-=== How do I uses the new high definition (HD) channels? ===
-Due to problems with our data sources, XMLTV (0.5.50 and before) and MythTV (0.20 and before) please follow the directions on the [wiki:HDTV] page.
-To make use of the HD flag, MythTV requires the setting of priorities.  In mythfrontend transverse the menus Utilities/Setup -> Setup-> TV Settings ->Recording Priorities -> Set Recording, and set HDTV Recording Priority = 2.  I also recommend enabling the Reschedule Higher Priorities option.  Then transverse Next -> Finish -> Channel Priorities, and select each of your HD channels and press left arrow, to decrease to -1.  Cancel exits.  This should tell MythTV to record flagged HD programs on a HD channel and non-flagged HD programs on a SD channel.
 === Some of my guide data looks wrong. How can I diagnose the problem? ===
 Because Shepherd employs many different grabbers, the first step is to figure out where the dodgy data came from. If you're interested in a particular time, you can use the "--ancestry" option to see how Shepherd put together guide data for a particular time. For example, to look at the ancestry of data for next Tuesday from 10:30pm - 11pm:
 {{{
 ~/.shepherd/tv_grab_au --ancestry "tuesday 10:30pm+30"
+tv_grab_au --ancestry "tuesday 10:30pm+30"
 }}}
 This will print out relevant guide data obtained during Shepherd's last successful run from each component. What you want to do is find the earliest point at which the timestamps are wrong.
 …
 }}}
 === Some shows have titles ALL IN CAPS, and often including unwanted info like "LIVE:" or "SEASON FINALE." ===
 You have EIT enabled in MythTV, which is overriding the guide data supplied by Shepherd. EIT is an over-the-air program guide broadcast by the TV stations themselves. In an ideal world, this would mean you don't need Shepherd at all. Unfortunately, so far EIT data seems to be very unreliable: show names change, times are inaccurate, descriptions are missing, etc. It is therefore recommended that you disable EIT in MythTV.
 As of September 2011, Shepherd will ask you to allow it to auto-disable EIT in MythTV during the --configure process. You can check or re-do this with:
+You have EIT enabled in MythTV, which is overriding the guide data supplied by Shepherd. EIT is an over-the-air program guide broadcast by the TV stations themselves. In an ideal world, this would mean you don't need Shepherd at all. Unfortunately, so far EIT data seems to be quite unreliable: show names change, times are inaccurate, descriptions are missing, etc. It is therefore recommended that you disable EIT in MythTV.
+Shepherd asks you to allow it to auto-disable EIT in MythTV during the --configure process. You can check or re-do this with:
 {{{
 tv_grab_au --configure-mythtv
 …
 Note that EIT can be enabled on a per-card and per-channel basis, so it's important to be turned off everywhere.
 === Show names keep changing! ===
 …
 In rare cases a show name can change simply because the datasource renames it and the new name is so different that Shepherd thinks it's a genuinely different show. (Subtle name changes, by contrast, are overridden by Shepherd to avoid breaking your recording rules.)
 === Some shows are always named an incorrect variant, e.g. "ABC News: Morning" instead of "ABC News" ===
 Shepherd's reconciler attempts to keep show names consistent, to allow you to reliably set recording rules based on show names. To do so, it may rename shows like "House, M.D." to "House" (if that's what it saw in the past), "Home and Away Catch-Up" to "Home and Away," and so on. This is necessary as show names frequently contain variations like these even within the same datasource; they also tend to vary from datasource to datasource.
 …
 Alternatively, there are Perl libraries for editing Storable files, including http://www.lothosoft.ch/thomas/perl-storableedit/ - using such a program, you can edit the hash under title_xlate_table for the title you no longer want to use set the 'translation' field to the new title you want.
+=== How do I make Shepherd use a particular grabber? ===
+Generally it's best to let Shepherd decide which grabbers to use, and in which order. One of its main benefits is its fault-tolerance: guide data sources tend to be fragile and can stop working unexpectedly, but Shepherd will work regardless. Relying on a particular grabber always being there is, unfortunately, not safe.
+Additionally, Shepherd makes reasonably intelligent decisions about which grabbers it would be best to use, and in which order.
+If you have a general preference for speed over quality, or vice versa, you can control this via [wiki:FAQ#CanImakeShepherdfaster the "--mode" option].
+To specify an exact order for grabbers, use the "--grabwith <grabber/s>" option. Shepherd will run the specified grabber(s) first, then others as needed to fill remaining holes in the data. For example:
+{{{
+tv_grab_au --grabwith oztivo,sbsnews_website
+}}}
 == MythTV-related ==
 …
 If you didn't answer "yes," or you're not sure it worked, re-do it. Most problems relating to integration between Shepherd and MythTV can be solved with this:
 {{{
 ~/.shepherd/shepherd --configure-mythtv
+tv_grab_au --configure-mythtv
 }}}
 Shepherd will run within a few minutes of you doing this. Please wait to see if it works by itself.
 …
 To see if Shepherd is currently running, look for "*** IN PROGRESS***" at the bottom of output from this command:
 {{{
 ~/.shepherd/shepherd --history
+tv_grab_au --history
 }}}
 …
 Other useful commands to check Shepherd's basic health:
 {{{
 ~/.shepherd/shepherd --status
 ~/.shepherd/shepherd --history
 ~/.shepherd/shepherd --check
 ~/.shepherd/shepherd --show-channels
+tv_grab_au --status
+tv_grab_au --history
+tv_grab_au --check
+tv_grab_au --show-channels
 }}}
 …
 === Shepherd is running hourly on my system. Shouldn't it run once per day? ===
 A default installation will setup an hourly cron job. This means Shepherd will run once per hour, but only download new guide data if it's been about a day since the last successful grab. Otherwise, it simply exits.
 If you prefer, you can edit your crontab to run Shepherd once per day at a time of your choice. However, the default install has two advantages over this method:
+. It works even if your system backend powers down when idle. A daily cron job, by contrast, requires your machine be on at that time every day.
 . If Shepherd fails for some reason (e.g. a temporary network problem), it will try again in an hour, rather than waiting a whole day.
+A default installation will create an hourly cron job, running Shepherd once per hour. Unless it's been a reasonable period of time since the last successful run, though, Shepherd will simply exit. Shepherd will resist running more frequently in order to avoid putting undue strain on TV guide datasources.
+If you prefer, you can edit your crontab to run Shepherd whenever you like. However, there's no downside to running Shepherd frequently (since it will just exit unless it's due for more data), whereas running Shepherd infrequently may lead to dwindling guide data if your system misses its run a few days in a row (e.g. due to powering off).
+=== How often does Shepherd compile new guide data? ===
+If it's been 22 hours since its last successful full run, Shepherd will attempt to gather guide data for the next 8 days. If it's been less than 22 hours but more than 4 hours, Shepherd will perform an "autorefresh", gathering data for just the current day.
 === How can I prevent mythfilldatabase adding unwanted channels to my video sources? ===
 …
 To negate this, you can use Shepherd's "--notquiet" option. You can make this a default by:
 {{{
 ~/.shepherd/tv_grab_au --component-set shepherd:notquiet
+tv_grab_au --component-set shepherd:notquiet
 }}}
 To disable this, use:
 {{{
 ~/.shepherd/shepherd --component-set shepherd
+tv_grab_au --component-set shepherd
 }}}
 …
 This will try to compare your Shepherd channels to your MythTV channels:
 {{{
 ~/.shepherd/shepherd --show-channels
+tv_grab_au --show-channels
 }}}
 If it doesn't look right, re-configure:
 {{{
 ~/.shepherd/shepherd --configure
+tv_grab_au --configure
 }}}
 If Shepherd is not able to access your MythTV (e.g. it's a remote backend), you will need to manually check that the [wiki:XMLTVIDs XMLTV IDs] you assigned to channels in Shepherd match those you assigned in MythTV. It doesn't matter what each XMLTV ID is, just that they match. For example, if in the ''mythtv-setup'' Channel Editor you have "ABC Melbourne" set to an XMLTV ID of "abc.free.au", you should also specify this as the XMLTV ID for the ABC channel in Shepherd. You can do this via --configure, as shown above.
 …
 Note that it can only do this if the 'augment_timezone' postprocessor can run which in turn requires access to your MythTV database.  Check that it is running with:
 {{{
 ~/.shepherd/shepherd --status
+tv_grab_au --status
 }}}
 and shows up as 'Enabled' and 'Ready' and has a status indicating it successfully processed data.