Shepherd FAQ

Here you can find answers to some of the most frequently asked questions about Shepherd... If you have a question not answered on this page, feel free to add the question into this page yourself (with no answer) - or, better yet, post the question to the mailing-list and then edit the question and answer into this Wiki page!

About Shepherd?

What is Shepherd?

Shepherd is an attempt to reconcile many different tv_grab_au scripts and make one cohesive reliable data set.

How does it work?

It works by calling a series of scripts that grab data from a large variety of sources, and then analysing the resulting XML data sets and determining which of the many is the most reliable. Postprocessors are used to augment the data sets with additional information (e.g. movie information from http://www.imdb.com, HDTV programming from http://www.dba.org.au etc.).

When switching between data sources, Shepherd's reconciler also tries to ensure that programme names are consistent. e.g. if you're used to recording a programme called "House" yet a different data source names it as "House, M.D.", Shepherd is smart enough to remember the original name and substitute it. No configuration is necessary to enable this; it happens automatically.

Shepherd is designed to be future proof, never requiring manual intervention once initially installed and configured. Shepherd will automatically update itself with fixes, enhancements and additional plugin components as and when they become available.

The shepherd_logic wiki page contains a more complete technical description of the various stages of Shepherd and how it works.

How can I get help or ask a few questions?

Feel free to join our mailing list by sending an email with "subscribe shepherd <email>" in the body to majordomo@interlink.com.au. Once you've joined, you can post to the list by emailing shepherd@interlink.com.au.

Note that there are no archives of the mailing list. If your question is not answered on this site, ask away...

Can I contribute?

Absolutely. Shepherd is a community project and is the result of countless contributors. If you wish to enhance some functionality within Shepherd (e.g. write a new postprocessor), implement some new fancy reconciling logic, implement a new grabber or just help out in answering questions or contributing to Wiki documentation, feel free to help!

Which operating systems does Shepherd support?

In theory, Shepherd (and its underlying components) will run on any operating system that supports Perl, as all scripts are currently written in Perl.

In practice, the developers all use Linux and MythTV, and that is what is known to work.

No effort has been put into making Shepherd work under Microsoft Windows or Windows Media Centre Edition, although it really shouldn't be too hard to get that working if anyone was motivated enough to do so.

Is Shepherd legal?

Some of the grabbers used by Shepherd read web sites that say they don't want their data used in PVRs, but that doesn't mean it's illegal. Shepherd doesn't copy or distribute data, but rather allows individuals at home to read it via their PVRs. It operates in the same manner as a browser, sending HTTP requests and formatting the resultant HTML for display in a manner appropriate to the user.

Installing Shepherd

How do I install Shepherd?

See the Installation page.

How important is it to install the optional Perl modules?

Some of Shepherd's grabbers require additional Perl modules to be installed, without which they won't function. They are listed as "optional" because Shepherd does not rely on any individual grabber to do its job; instead it draws on as many or as few of its available grabbers as necessary to acquire guide data for the time period and channels you want.

Sometimes Shepherd can do this with a single grabber. More commonly, it employees multiple grabbers and combines their results.

Generally speaking, Shepherd can perform very well even if some of its grabbers are disabled or unsupported (i.e. missing modules). However, it will probably perform more efficiently, reliably, and possibly more accurately if you can enable all of its grabbers.

Using Shepherd

How well is Shepherd working?

A summary of Shepherd's performance can be viewed by:

~/.shepherd/tv_grab_au --status

In particular, note the last line, which tells you the percentage of wanted data it acquired. If it's less than 100.00%, Shepherd wasn't able to completely data for all your channels over the next 7 days. If you haven't enabled all of Shepherd's grabbers, you will probably benefit from doing this.

If Shepherd is grabbing 100% of wanted data, then enabling additional grabbers may be unnecessary. However, doing so will still improve Shepherd's ability to tolerate a grabber failure, may allow it to run faster and use less bandwidth, and may improve its data quality.

Is there a log file somewhere?

Yes, in the log/ subdirectory of your Shepherd installation (usually ~/.shepherd/log/).

How do I make Shepherd use a particular grabber?

Generally, it's best to let Shepherd make its own decisions about which grabbers to use, and in which order. By default Shepherd operates in "Efficiency" mode, whereby it selects grabbers based on maximizing data quantity and quality while minimizing time and bandwidth.

If you wish, however, you may run Shepherd in "Quality" or "Speed" mode. In Quality mode, Shepherd will favour grabbers that tend to deliver the most reliable, highest-quality data, even if they take a long time. (And be aware: some grabbers can take a very long time: a week of data may take 6 hours or more.) In Speed mode, Shepherd will favour grabbers that complete quickly, even if their data isn't the highest-quality. (Although note mode only affects grabbers: other components, such as postprocessors, may still be slow.)

In all modes, Shepherd will employ as many grabbers as necessary to completely fill the next 7 days with data.

To run Shepherd once in a particular mode:

~/.shepherd/shepherd --mode quality

or to permanently set Shepherd in a particular mode:

~/.shepherd/shepherd --component-set shepherd:mode=speed

To specify an exact order for grabbers, regardless of whether they have any data that Shepherd wants, or what its speed or quality is, you can use the "--grabwith <grabber/s>" option. Shepherd will run the specified grabber(s) first, then any others as needed to fill remaining holes in the data. For example:

~/.shepherd/tv_grab_au --grabwith oztivo,sbsnews_website

Can I set some default options?

If you want Shepherd to always be called as if it was sent a particular command-line option, you can use:

~/.shepherd/tv_grab_au --component-set shepherd:<option>

For example, this would make Shepherd always run as if called with the option "--grabwith=abc_website":

~/.shepherd/tv_grab_au --component-set shepherd:grabwith=abc_website

If you want to add multiple options they all need to be set with one command otherwise the final command will override any previously set commands. For example to add both "--notquiet" and "--grabwith=abc_website" :

~/.shepherd/tv_grab_au --component-set shepherd:notquiet:grabwith=abc_website

You can also set default options for any component, e.g.:

~/.shepherd/tv_grab_au --component-set abc_website:do-extra-days

To clear all default options, call --component-set with no argument. E.g.:

~/.shepherd/tv_grab_au --component-set shepherd

Can I grab more than 7 days of data?

Yes, although it's a little kludgy at the moment. Some grabbers / data sources support more than 7 days of data: abc_website offers up to 28 days of ABC & ABC2, while southerncross_website can do 7-18 days of Southern Cross/TEN in some regions.

You can enable this via the component setting:

~/.shepherd/tv_grab_au --component-set abc_website:do-extra-days

This instructs the abc_website grabber to fetch extra days. However, it does not guarantee that Shepherd will ever call that grabber. If the extra days is especially important to you, you could also ensure that abc_website is always run:

~/.shepherd/tv_grab_au --component-set shepherd:grabwith=abc_website

Note that as mentioned above, this restricts Shepherd's ability to fetch the best possible data in the most efficient way.

You can disable the setting by:

~/.shepherd/tv_grab_au --component-set abc_website

Can I specify different configuration files for Shepherd to use?

No. Shepherd always expects to use the same configuration file (usually ~/.shepherd/shepherd.conf).

You can use tricks like changing the environment variable HOME, to run multiple installs of shepherd. But it's not very efficient, in that it will lead to a fair bit of redundant downloading, but otherwise it is a good solution for someone wanting data from multiple regions.

HOME=/first/directory /first/directory/.shepherd/shepherd

HOME=/second/directory /second/directory/.shepherd/shepherd

Another similar way is to 'mv' the .shepherd directory.

mv ~/.shepherd ~/.shepherd.1
mv ~/.shepherd.2 ~/.shepherd
~/.shepherd/shepherd

mv ~/.shepherd ~/.shepherd.2
mv ~/.shepherd.1 ~/.shepherd
~/.shepherd/shepherd

My high definition (HD) channels are missing programs?

You must configure standard definition (SD) channels for the corresponding HD channels for them to be populated correctly.

It is now possible to obtain fully populated HD channels with KNOWN HD programs flagged as HD. This is the DEFAULT for new installs. To enable on old installs, go to your shepherd install and execute:

rm ~/.shepherd/postprocessors/flag_aus_hdtv/flag_aus_hdtv.config

Then see How do I uses the new high definition (HD) channels?

Originally the HD channels were only populated with KNOWN HD programs. The idea is to have both SD and HD, and increase the priority of HD channels, so programs record as HD when available. To change to this behaviour use these commands:

cd ~/.shepherd/postprocessors/flag_aus_hdtv
ln -s ../../references/Shepherd
./flag_aus_hdtv --set=action:copy
rm Shepherd

WARNING: Some stations do upscaling from SD to HD; you could record the HD version but the SD version is at least half the size for the same detail. Also some stations run a program of scenery in a loop; if you recorded their HD channel you would miss your program.

Another way to obtain fully populated HD channels, is use the SD xmlids for the HD channels and remove your HD channels from shepherd BUT you will miss any SD channel and HD channel divergence.

If any additional programs should be flagged HD, please let us known on our mail list.

How do I uses the new high definition (HD) channels?

Due to problems with our data sources, XMLTV (0.5.50 and before) and MythTV (0.20 and before) please follow the directions on the HDTV page.

To make use of the HD flag, MythTV requires the setting of priorities. In mythfrontend transverse the menus Utilities/Setup -> Setup-> TV Settings ->Recording Priorities -> Set Recording, and set HDTV Recording Priority = 2. I also recommend enabling the Reschedule Higher Priorities option. Then transverse Next -> Finish -> Channel Priorities, and select each of your HD channels and press left arrow, to decrease to -1. Cancel exits. This should tell MythTV to record flagged HD programs on a HD channel and non-flagged HD programs on a SD channel.

Some of my guide data looks wrong. How can I diagnose the problem?

Because Shepherd employs many different grabbers, the first step is to figure out where the dodgy data came from. If you're interested in a particular time, you can use the "--ancestry" option to see how Shepherd put together guide data for a particular time. For example, to look at the ancestry of data for next Tuesday from 10:30pm - 11pm:

~/.shepherd/tv_grab_au --ancestry "tuesday 10:30pm+30"

This will print out relevant guide data obtained during Shepherd's last successful run from each component. What you want to do is find the earliest point at which the timestamps are wrong.

• If the data looks wrong in a grabber, it's either a problem with the grabber itself or the grabber's data source.
• If the data looks fine in the grabber, but is bad in the output of a reconciler, postprocessor, or Shepherd itself, it's a Shepherd problem.

Either way, armed with this information you should be able to get further help from the mailing list.

The --ancestry option requires the Perl module File::Find::Rule. You can install it with the command:

sudo cpan File::Find::Rule

or, for Debian-based distributions:

sudo apt-get install libfile-find-rule-perl

MythTV-related

Shepherd works when I run it from the command-line, but not automatically via mythfilldatabase.

Make sure you've created the symbolic link from tv_grab_au to Shepherd, and that the following command points to your Shepherd executable:

ls -l `which tv_grab_au`

If it points somewhere else--for example, if you have an old tv_grab_au script from a previous grabber in /usr/local/bin/--you should delete/replace this with the symlink to Shepherd.

Further information may be gleaned by running mythfilldatabase from the command-line and looking for any error messages between the "-----Start/End of XMLTV output----" lines.

When mythfilldatabase runs (SVN release of mythtv, mythtv 0.21 and later), I can't see what Shepherd is doing

Starting with MythTV 0.21 and SVN release around February 2007, mythfilldatabase (by default) adds '--quiet' onto the command-line when calling the tv_grab_au script. To negate this, you can use Shepherd's "--notquiet" option. You can make this a default by:

~/.shepherd/tv_grab_au --component-set shepherd:notquiet

To disable this, use:

~/.shepherd/tv_grab_au --component-set shepherd

How can I prevent mythfilldatabase adding unwanted channels to my video sources?

Make sure mythfilldatabase (not Shepherd) is invoked with the "--update" option, so it will not add any missing channels to your video sources. (This can be an issue if you have video sources that receive different sets of channels, for example free tv and pay tv.)

My guide data is listed against the wrong channels inside MythTV.

It's important that the XMLTV IDs you assigned to channels in Shepherd match those you assigned in MythTV. It doesn't matter what each XMLTV ID is, just that they match. For example, if in the mythtv-setup Channel Editor you have "ABC Melbourne" set to an XMLTV ID of "abc.free.au", you should also specify this as the XMLTV ID for the ABC channel in Shepherd:

tv_grab_au --configure

Which timezone should I set MythTV to? Auto, None, +1000?

Usually it doesn't matter. Shepherd will look up MythTV's timezone setting, compare this to your system clock, and ensure everything lines up.

If you're running Shepherd on a different box to MythTV, though, set your MythTV timezone to "None."

Shepherd assumes that your machine's timezone has been set correctly for the region you've selected. To check this is the case, make sure these commands both display the correct time and timezone for the region you're using Shepherd for:

perl -e 'use POSIX; print POSIX::strftime("%z %x %X %Z\n", localtime(time));'
date "+%z %x %X %Z"

If you need to set your machine's timezone, Debian-based distributions can use the 'tzconfig' command (run as root). The TZ environment variable could also be set to correct any differences. Add in ~/.profile (to make it user specific), /etc/profile (to make it machine wide), or just before you execute shepherd. eg:

TZ="Australia/Brisbane" 
export TZ

You will need to log off and back in to pick up the change.

Note: A common timezone problem (unrelated to Shepherd) is when times are out in MythWeb?, but correct in the rest of MythTV. This is because PHP5, unlike PHP4, requires you to explicitly set a timezone in php.ini:

In a terminal, type:

sudo gedit /etc/php5/apache2/php.ini

Within gedit, Search > Go to line > 603. Change the line from:

;date.timezone =

to

date.timezone = Australia/Adelaide

... or whatever is appropriate for your location. Make sure to remove the semi-colon (comment)!

Save, exit gedit, and restart apache:

sudo /etc/init.d/apache2 restart

Your MythWeb? times should now match what is displayed within MythTV.