Changes between Version 71 and Version 72 of FAQ


Ignore:
Timestamp:
09/30/13 18:40:06 (3 years ago)
Author:
max
Comment:

update!

Legend:

Unmodified
Added
Removed
Modified
  • FAQ

    v71 v72  
    1111=== What is Shepherd? === 
    1212 
    13 Shepherd is an attempt to reconcile many different tv_grab_au scripts and make one cohesive reliable data set. 
     13Shepherd is software for compiling an Electronic Program Guide (EPG), by managing a flock of guide data grabbers and postprocessors. 
    1414 
    1515=== How does it work? === 
     
    2121Shepherd is designed to be ''future proof'', never requiring manual intervention once initially installed and configured.  Shepherd will automatically update itself with fixes, enhancements and additional ''plugin'' components as and when they become available. 
    2222 
     23By default, Shepherd performs one full run per day, gathering guide data for the next 8 days. It also autorefreshes the current day's data once every four hours or so, to catch for last-minute schedule changes. 
     24 
    2325The [wiki:shepherd_logic] wiki page contains a more complete technical description of the various stages of Shepherd and how it works. 
    2426 
     
    3638In theory, Shepherd (and its underlying components) will run on any operating system that supports Perl, as all scripts are currently written in Perl. 
    3739 
    38 In practice, the developers all use Linux and MythTV, and that is what is known to work. 
    39  
    40 No effort has been put into making Shepherd work under Microsoft Windows or Windows Media Centre Edition, although it really shouldn't be too hard to get that working if anyone was motivated enough to do so. 
     40In practice, the developers all use Linux, so that's what is tested. There are Shepherd users running on [wiki:ShepherdForWindows Windows] and [wiki:Installation#AppleOSX10.8 OS X] but these require more fiddling. 
     41 
     42=== Does Shepherd only work with MythTV? === 
     43 
     44Shepherd will work with any PVR-type software capable of reading XMLTV (which is most of them). MythTV users get a relatively painless installation, since Shepherd can set itself up automatically, but MythTV is not a pre-requisite. Shepherd is often used on alternatives, such as Freevo. 
     45 
     46If you are having trouble getting Shepherd to work outside the usual environment of Linux + MythTV, don't hesitate to write in to our mailing list for help. 
    4147 
    4248=== Is Shepherd legal? === 
    4349 
    44 Some of the grabbers used by Shepherd read web sites that say they don't want their data used in PVRs, but that [http://www.cse.unsw.edu.au/~willu/xmltv/index.html#Legal doesn't mean it's illegal]. Shepherd doesn't copy or distribute data, but rather allows individuals at home to read it via their PVRs. It operates in the same manner as a browser, sending HTTP requests and formatting the resultant HTML for display in a manner appropriate to the user. 
     50Some of the grabbers used by Shepherd read web sites that say they don't want their data used in PVRs, but that [http://www.cse.unsw.edu.au/~willu/xmltv/index.html#Legal doesn't mean it's illegal]. Shepherd doesn't host or distribute data, but rather allows individuals at home to access it via their PVRs. Like a browser, it sends HTTP requests to public websites and formats the result for display in a manner appropriate to the user. 
    4551 
    4652== Installing Shepherd == 
     
    6773A summary of Shepherd's performance can be viewed by: 
    6874{{{ 
    69 ~/.shepherd/tv_grab_au --status 
     75tv_grab_au --status 
    7076}}} 
    7177In particular, note the last line, which tells you the percentage of wanted data it acquired. If it's less than 100.00%, Shepherd wasn't able to completely data for all your channels over the next 7 days. If you haven't enabled all of Shepherd's grabbers, you will probably benefit from doing this. 
     
    7884 
    7985 
    80 === How do I make Shepherd use a particular grabber? === 
    81 Generally it's best to let Shepherd decide which grabbers to use, and in which order. One of its main benefits is its fault-tolerance: guide data sources tend to be fragile and can stop working unexpectedly, but Shepherd will work regardless. Relying on a particular grabber always being there is, unfortunately, not safe. 
    82  
    83 If you have a general preference for speed over quality, or vice versa, you can control this via [wiki:FAQ#CanImakeShepherdfaster the "--mode" option]. 
    84  
    85 To specify an exact order for grabbers, use the "--grabwith <grabber/s>" option. Shepherd will run the specified grabber(s) first, then others as needed to fill remaining holes in the data. For example: 
    86 {{{ 
    87 ~/.shepherd/tv_grab_au --grabwith oztivo,sbsnews_website 
    88 }}} 
     86=== What is autorefresh? === 
     87When it's been more than 4 hours but less than 22 hours since the last run, Shepherd will switch into autorefresh mode, gathering data for the current day only. It will output the result via STDOUT, as usual, and create the file ~/.shepherd/refresh.xmltv. This allows Shepherd to catch last-minute schedule changes. It tends to complete very quickly (for Shepherd), since even the slower grabbers are usually able to quickly verify if there have been any time or title changes since they last ran. 
     88 
     89If you don't wish Shepherd to enter autorefresh mode, send the --noautorefresh option. 
     90 
     91Shepherd will '''not''' enter autorefresh mode if you specify a number of --days, since this would require overriding your preference. But if you are happy for Shepherd to override it during autorefreshes only, send --allowautorefresh. 
     92 
     93Autorefresh does not touch the default output.xmltv file, so functions such as '''--refill-mythtv''' will use your last full successful run, not any subsequent autorefresh. You may wish to override this if you want to be able to rely on output.xmltv always containing the latest data, regardless of whether it's a single day's data or a week's. If so, use '''--output'''. For example, to always output into output.xmltv, even autorefreshes: 
     94{{{ 
     95tv_grab_au --output ~/.shepherd/output.xmltv 
     96}}} 
     97In that case, you would probably want to set it as a default option, i.e.: 
     98{{{ 
     99tv_grab_au --component-set shepherd:output=output.xmltv 
     100}}} 
     101Or to refill MythTV will the latest autorefresh data, rather than the latest full run: 
     102{{{ 
     103tv_grab_au --refill-mythtv --output ~/.shepherd/refresh.xmltv 
     104}}} 
     105 
    89106 
    90107=== Why is Shepherd so slow? === 
     
    93110Since Shepherd runs in the background, how long it takes makes no difference. If you'd like it to complete faster, though, you may use [wiki:FAQ#CanImakeShepherdfaster the "--mode" option]. 
    94111 
     112 
    95113=== Can I make Shepherd faster?  === 
    96114By default Shepherd operates in "Quality" mode, whereby it selects grabbers based on maximizing data quantity and quality. This mode can take a long time: around 15-30 minutes under normal operating conditions, and several hours if it's being run for the first time, with no cache and needing to fill a full week.  
     
    102120To run Shepherd once in a particular mode: 
    103121{{{ 
    104 ~/.shepherd/shepherd --mode speed 
     122tv_grab_au --mode speed 
    105123}}} 
    106124or to permanently set Shepherd in a particular mode: 
    107125{{{ 
    108 ~/.shepherd/shepherd --component-set shepherd:mode=speed 
     126tv_grab_au --component-set shepherd:mode=speed 
    109127}}} 
    110128 
     
    113131If you want Shepherd to always be called as if it was sent a particular command-line option, you can use: 
    114132{{{ 
    115 ~/.shepherd/tv_grab_au --component-set shepherd:<option> 
     133tv_grab_au --component-set shepherd:<option> 
    116134}}} 
    117135For example, this would make Shepherd always run as if called with the option "--grabwith=abc_website": 
    118136{{{ 
    119 ~/.shepherd/tv_grab_au --component-set shepherd:grabwith=abc_website 
     137tv_grab_au --component-set shepherd:grabwith=abc_website 
    120138}}} 
    121139If you want to add multiple options they all need to be set with one command otherwise the final command will override any previously set commands. For example to add both "--notquiet" and "--grabwith=abc_website" : 
    122140{{{ 
    123 ~/.shepherd/tv_grab_au --component-set shepherd:notquiet:grabwith=abc_website 
     141tv_grab_au --component-set shepherd:notquiet:grabwith=abc_website 
    124142}}} 
    125143You can also set default options for any component, e.g.: 
    126144{{{ 
    127 ~/.shepherd/tv_grab_au --component-set abc_website:do-extra-days 
     145tv_grab_au --component-set abc_website:do-extra-days 
    128146}}} 
    129147To clear all default options, call --component-set with no argument. E.g.: 
    130148{{{ 
    131 ~/.shepherd/tv_grab_au --component-set shepherd 
    132 }}} 
     149tv_grab_au --component-set shepherd 
     150}}} 
     151 
     152 
    133153=== Can I grab more than 7 days of data? === 
    134 (Note: the Shepherd default is now eight days.) 
     154(Note: The Shepherd default is now eight days.) 
    135155 
    136156You can ask Shepherd to try to grab however many days you like. Some channels in some regions offer up to 28 days of data; others as few as three or four. Generally, you can get at least 7 days for all but the community channels, and 14+ days for ABC and SBS. 
     
    147167}}} 
    148168 
     169'''Note:''' This will disable autorefresh, which is where Shepherd refreshes the current day's data once every 4 hours, unless you also specify --allowautorefresh. 
     170 
     171 
    149172=== Can I get guide data from multiple different regions? === 
    150173 
     
    162185 
    163186Users grabbing data from regions in different timezones should also ensure that each Shepherd instance has [wiki:FAQ#WhichtimezoneshouldIsetMythTVtoAutoNone1000 the correct TZ variable set]. 
     187 
    164188 
    165189=== Can I specify different configuration files for Shepherd to use? === 
     
    187211Multiple versions of Shepherd will produce multiple output.xmltv files, which you will need to pass to your PVR. In MythTV, for example, you should not use the default cron job (which invokes mythfilldatabase rather than Shepherd), but rather create your own cron jobs to run each instance of Shepherd, then feed their output to MythTV [wiki:Installation#Multiplesources as described here]. 
    188212 
    189 === My high definition (HD) channels are missing programs? === 
    190  
    191 You must configure standard definition (SD) channels for the corresponding HD channels for them to be populated correctly. 
    192  
    193 It is now possible to obtain fully populated HD channels with KNOWN HD programs flagged as HD.  This is the DEFAULT for new installs.  To enable on old installs, go to your shepherd install and execute: 
    194 {{{ 
    195 rm ~/.shepherd/postprocessors/flag_aus_hdtv/flag_aus_hdtv.config 
    196 }}} 
    197 Then see [http://svn.whuffy.com/wiki/FAQ#HowdoIusesthenewhighdefinitionHDchannels How do I use the new high definition (HD) channels?] 
    198  
    199 Originally the HD channels were only populated with KNOWN HD programs.  The idea is to have both SD and HD, and increase the priority of HD channels, so programs record as HD when available.  To change to this behaviour use these commands: 
    200 {{{ 
    201 cd ~/.shepherd/postprocessors/flag_aus_hdtv 
    202 ln -s ../../references/Shepherd 
    203 ./flag_aus_hdtv --set=action:copy 
    204 rm Shepherd 
    205 }}} 
    206  
    207 WARNING: Some stations do upscaling from SD to HD; you could record the HD version but the SD version is at least half the size for the same detail.  Also some stations run a program of scenery in a loop; if you recorded their HD channel you would miss your program. 
    208  
    209 Another way to obtain fully populated HD channels, is use the SD xmlids for the HD channels and remove your HD channels from shepherd BUT you will miss any SD channel and HD channel divergence. 
    210  
    211 If any additional programs should be flagged HD, please let us known on our mail list. 
    212  
    213  
    214 === How do I uses the new high definition (HD) channels? === 
    215  
    216 Due to problems with our data sources, XMLTV (0.5.50 and before) and MythTV (0.20 and before) please follow the directions on the [wiki:HDTV] page. 
    217  
    218 To make use of the HD flag, MythTV requires the setting of priorities.  In mythfrontend transverse the menus Utilities/Setup -> Setup-> TV Settings ->Recording Priorities -> Set Recording, and set HDTV Recording Priority = 2.  I also recommend enabling the Reschedule Higher Priorities option.  Then transverse Next -> Finish -> Channel Priorities, and select each of your HD channels and press left arrow, to decrease to -1.  Cancel exits.  This should tell MythTV to record flagged HD programs on a HD channel and non-flagged HD programs on a SD channel. 
    219  
    220213=== Some of my guide data looks wrong. How can I diagnose the problem? === 
    221214 
    222215Because Shepherd employs many different grabbers, the first step is to figure out where the dodgy data came from. If you're interested in a particular time, you can use the "--ancestry" option to see how Shepherd put together guide data for a particular time. For example, to look at the ancestry of data for next Tuesday from 10:30pm - 11pm: 
    223216{{{ 
    224 ~/.shepherd/tv_grab_au --ancestry "tuesday 10:30pm+30" 
     217tv_grab_au --ancestry "tuesday 10:30pm+30" 
    225218}}} 
    226219This will print out relevant guide data obtained during Shepherd's last successful run from each component. What you want to do is find the earliest point at which the timestamps are wrong. 
     
    240233}}} 
    241234 
     235 
    242236=== Some shows have titles ALL IN CAPS, and often including unwanted info like "LIVE:" or "SEASON FINALE." === 
    243 You have EIT enabled in MythTV, which is overriding the guide data supplied by Shepherd. EIT is an over-the-air program guide broadcast by the TV stations themselves. In an ideal world, this would mean you don't need Shepherd at all. Unfortunately, so far EIT data seems to be very unreliable: show names change, times are inaccurate, descriptions are missing, etc. It is therefore recommended that you disable EIT in MythTV. 
    244  
    245 As of September 2011, Shepherd will ask you to allow it to auto-disable EIT in MythTV during the --configure process. You can check or re-do this with: 
     237You have EIT enabled in MythTV, which is overriding the guide data supplied by Shepherd. EIT is an over-the-air program guide broadcast by the TV stations themselves. In an ideal world, this would mean you don't need Shepherd at all. Unfortunately, so far EIT data seems to be quite unreliable: show names change, times are inaccurate, descriptions are missing, etc. It is therefore recommended that you disable EIT in MythTV. 
     238 
     239Shepherd asks you to allow it to auto-disable EIT in MythTV during the --configure process. You can check or re-do this with: 
    246240{{{ 
    247241tv_grab_au --configure-mythtv 
     
    249243 
    250244Note that EIT can be enabled on a per-card and per-channel basis, so it's important to be turned off everywhere. 
     245 
    251246 
    252247=== Show names keep changing! === 
     
    255250In rare cases a show name can change simply because the datasource renames it and the new name is so different that Shepherd thinks it's a genuinely different show. (Subtle name changes, by contrast, are overridden by Shepherd to avoid breaking your recording rules.) 
    256251 
     252 
    257253=== Some shows are always named an incorrect variant, e.g. "ABC News: Morning" instead of "ABC News" === 
    258254Shepherd's reconciler attempts to keep show names consistent, to allow you to reliably set recording rules based on show names. To do so, it may rename shows like "House, M.D." to "House" (if that's what it saw in the past), "Home and Away Catch-Up" to "Home and Away," and so on. This is necessary as show names frequently contain variations like these even within the same datasource; they also tend to vary from datasource to datasource. 
     
    268264Alternatively, there are Perl libraries for editing Storable files, including http://www.lothosoft.ch/thomas/perl-storableedit/ - using such a program, you can edit the hash under title_xlate_table for the title you no longer want to use set the 'translation' field to the new title you want.  
    269265 
     266 
     267=== How do I make Shepherd use a particular grabber? === 
     268Generally it's best to let Shepherd decide which grabbers to use, and in which order. One of its main benefits is its fault-tolerance: guide data sources tend to be fragile and can stop working unexpectedly, but Shepherd will work regardless. Relying on a particular grabber always being there is, unfortunately, not safe. 
     269 
     270Additionally, Shepherd makes reasonably intelligent decisions about which grabbers it would be best to use, and in which order. 
     271 
     272If you have a general preference for speed over quality, or vice versa, you can control this via [wiki:FAQ#CanImakeShepherdfaster the "--mode" option]. 
     273 
     274To specify an exact order for grabbers, use the "--grabwith <grabber/s>" option. Shepherd will run the specified grabber(s) first, then others as needed to fill remaining holes in the data. For example: 
     275{{{ 
     276tv_grab_au --grabwith oztivo,sbsnews_website 
     277}}} 
     278 
     279 
    270280== MythTV-related == 
    271281 
     
    280290If you didn't answer "yes," or you're not sure it worked, re-do it. Most problems relating to integration between Shepherd and MythTV can be solved with this: 
    281291{{{ 
    282 ~/.shepherd/shepherd --configure-mythtv 
     292tv_grab_au --configure-mythtv 
    283293}}} 
    284294Shepherd will run within a few minutes of you doing this. Please wait to see if it works by itself. 
     
    286296To see if Shepherd is currently running, look for "*** IN PROGRESS***" at the bottom of output from this command: 
    287297{{{ 
    288 ~/.shepherd/shepherd --history 
     298tv_grab_au --history 
    289299}}} 
    290300 
     
    310320Other useful commands to check Shepherd's basic health: 
    311321{{{ 
    312 ~/.shepherd/shepherd --status 
    313 ~/.shepherd/shepherd --history 
    314 ~/.shepherd/shepherd --check 
    315 ~/.shepherd/shepherd --show-channels 
     322tv_grab_au --status 
     323tv_grab_au --history 
     324tv_grab_au --check 
     325tv_grab_au --show-channels 
    316326}}} 
    317327 
     
    374384 
    375385=== Shepherd is running hourly on my system. Shouldn't it run once per day? === 
    376 A default installation will setup an hourly cron job. This means Shepherd will run once per hour, but only download new guide data if it's been about a day since the last successful grab. Otherwise, it simply exits. 
    377  
    378 If you prefer, you can edit your crontab to run Shepherd once per day at a time of your choice. However, the default install has two advantages over this method: 
    379  
    380  1. It works even if your system backend powers down when idle. A daily cron job, by contrast, requires your machine be on at that time every day. 
    381  1. If Shepherd fails for some reason (e.g. a temporary network problem), it will try again in an hour, rather than waiting a whole day. 
     386A default installation will create an hourly cron job, running Shepherd once per hour. Unless it's been a reasonable period of time since the last successful run, though, Shepherd will simply exit. Shepherd will resist running more frequently in order to avoid putting undue strain on TV guide datasources. 
     387 
     388If you prefer, you can edit your crontab to run Shepherd whenever you like. However, there's no downside to running Shepherd frequently (since it will just exit unless it's due for more data), whereas running Shepherd infrequently may lead to dwindling guide data if your system misses its run a few days in a row (e.g. due to powering off). 
     389 
     390=== How often does Shepherd compile new guide data? === 
     391If it's been 22 hours since its last successful full run, Shepherd will attempt to gather guide data for the next 8 days. If it's been less than 22 hours but more than 4 hours, Shepherd will perform an "autorefresh", gathering data for just the current day. 
    382392 
    383393=== How can I prevent mythfilldatabase adding unwanted channels to my video sources? === 
     
    398408To negate this, you can use Shepherd's "--notquiet" option. You can make this a default by: 
    399409{{{ 
    400 ~/.shepherd/tv_grab_au --component-set shepherd:notquiet 
     410tv_grab_au --component-set shepherd:notquiet 
    401411}}} 
    402412 
    403413To disable this, use: 
    404414{{{ 
    405 ~/.shepherd/shepherd --component-set shepherd 
     415tv_grab_au --component-set shepherd 
    406416}}} 
    407417 
     
    409419This will try to compare your Shepherd channels to your MythTV channels: 
    410420{{{ 
    411 ~/.shepherd/shepherd --show-channels 
     421tv_grab_au --show-channels 
    412422}}} 
    413423If it doesn't look right, re-configure: 
    414424{{{ 
    415 ~/.shepherd/shepherd --configure 
     425tv_grab_au --configure 
    416426}}} 
    417427If Shepherd is not able to access your MythTV (e.g. it's a remote backend), you will need to manually check that the [wiki:XMLTVIDs XMLTV IDs] you assigned to channels in Shepherd match those you assigned in MythTV. It doesn't matter what each XMLTV ID is, just that they match. For example, if in the ''mythtv-setup'' Channel Editor you have "ABC Melbourne" set to an XMLTV ID of "abc.free.au", you should also specify this as the XMLTV ID for the ABC channel in Shepherd. You can do this via --configure, as shown above. 
     
    433443Note that it can only do this if the 'augment_timezone' postprocessor can run which in turn requires access to your MythTV database.  Check that it is running with: 
    434444{{{ 
    435 ~/.shepherd/shepherd --status 
     445tv_grab_au --status 
    436446}}} 
    437447and shows up as 'Enabled' and 'Ready' and has a status indicating it successfully processed data.