Installation

Shepherd should be installed as whichever user is going to run it. We recommend that this not be root.

1. [MythTV Users only] Set up your MythTV channels by running mythtv-setup, going into the Channel Editor, and using the Channel Scanner.
2. [optional] Set a proxy if needed:

   export http_proxy=http://192.168.1.250:8888/
   

3. Download Shepherd:

   wget http://www.whuffy.com/shepherd/shepherd
   

4. Run it:

   perl shepherd
   

5. If Shepherd won't run because a "mandatory module" is not found, install it (see "Perl Dependencies") and try again. You may have to install several mandatory Perl modules before Shepherd can run.
   
   When Shepherd runs for the first time, it will attempt to download and install components (probably into ~/.shepherd/). Most likely, one or more components will fail to install due to missing Perl modules, but ignore this for now.
6. Answer the configuration questions to select your region. When asked if you want Guided Channel Selection, say "yes". Shepherd will then step through each of your MythTV channels, asking you to choose appropriate guide data for each one. Some MythTV channels you won't want guide data for (e.g. radio channels, guide channels, duplicate channels); that's fine.
   
   If you are not a MythTV user, or Shepherd cannot access your MythTV, you will need to use Advanced Channel Selection instead to manually enter XMLTV IDs for each channel.
   
   When asked to confirm that you want to create Shepherd's configuration file, say yes.
7. [optional] Shepherd will now test all its components. If any require configuration (e.g. the oztivo grabber), you will be prompted to do this. You do not have to configure components, and you can do it later: just press enter to skip any questions asked of you.
8. [optional] Shepherd will ask if you want to grab data now. You probably don't, because you haven't installed the optional Perl modules, so say no. Shepherd will exit.
9. [optional] For maximum functionality, install any missing Perl modules required by components. See the detailed instructions and why this is worth doing. Essentially, you want to install any missing Perl modules Shepherd complains about when you do this:

   ~/.shepherd/shepherd --check
   

10. [optional] Symlink /usr/bin/tv_grab_au to Shepherd. This enables mythfilldatabase to call Shepherd automatically. Also, it's sometimes convenient to invoke Shepherd with a simple tv_grab_au command:

    sudo ln -s ~/.shepherd/shepherd /usr/bin/tv_grab_au
    

11. [optional] You may delete the original Shepherd file you downloaded (leaving the newly installed version in ~/.shepherd/).
12. [optional] You can test Shepherd (relatively) quickly by asking it to gather data for just today. At the end of this command, you will have guide data in ~/.shepherd/output.xmltv:

    ~/.shepherd/shepherd --days 1
    

13. [optional] If you'd like Shepherd to provide you with some channel icons, set these up:

    ~/.shepherd/shepherd --set-icons
    

14. [optional] Install Tor if you wish to increase the speed of some grabbers. (Note: currently no grabbers use Tor, so the benefit of this step is marginal.)

Integration with MythTV

Shepherd's job is to create a file of TV guide data for the next 7 days: by default, ~/.shepherd/output.xmltv. (It also creates a log file at ~/.shepherd/log/shepherd.log.) Most users want this guide data to end up inside MythTV. This task is handled by MythTV's mythfilldatabase program.

To periodically run Shepherd, you can:

1. In mythtv-setup -> Video Sources, select "Australia" (tv_grab_au). Upon exiting this page, MythTV will run Shepherd in configuration mode. Control-C to exit. Run the MythTV frontend, and in Utilities/Setup -> Setup -> General -> Mythfilldatabase, check "Automatically run mythfilldatabase". You might want to tweak the Start and End times (if convenient, please avoid 2AM and 8AM, as these tend to be peak times), but otherwise all default values should be fine. Note: This method will run Shepherd as whichever user mythfilldatabase runs as. If mythfilldatabase runs as root, and you cannot change that, then in order for Shepherd to find its config files, it must be installed as root, too (despite this being a heightened security risk).
   
   or
   
   
2. Create a cron job to run Shepherd yourself, and have mythfilldatabase read Shepherd's output with the --file option.

You may also wish to ensure that your Shepherd and MythTV channels are correctly mapped to each other:

~/.shepherd/shepherd --show-channels

Integration With Freevo

Freevo will use the Shepherd data with a little alteration. In local_conf.py set the XMLTV_GRABBER XMLTV_GRABBER = '/home/tv/.shepherd/tv_grab_au --timeoffset=Auto' (or /usr/bin/tv_grab_au if you have set the symlink) If you have existing xmltv channel id set, use the ones you have already set when running

~/.shepherd/shepherd --configure

Otherwise use any channel id you choose. Test your setup with

freevo tv_grab

Some Useful Commands

A full list of command-line options:

~/.shepherd/shepherd --help

The status screen:

~/.shepherd/shepherd --status

Re-configure Shepherd (preserving old settings except where overridden):

~/.shepherd/shepherd --configure

Check whether any components are missing Perl modules or need to be configured:

~/.shepherd/shepherd --check

Check your channel mappings:

~/.shepherd/shepherd --show-channels

Perl Dependencies

Shepherd and its components rely on various third-party Perl modules. Some are mandatory, meaning that Shepherd won't run without them. Most, however, are optional, meaning that Shepherd can still operate without them, but some of its components (including grabbers) won't.

You should install all the optional modules if possible, as this increases Shepherd's efficiency, reliability, and possibly the quality of the guide data it delivers. For more information, see the FAQ.

To see which Perl modules you're missing, do this:

~/.shepherd/shepherd --check

If applicable, follow the distribution-specific instructions for Debian-based, Fedora Core 6, and Gentoo. Alternately you can use the general method.

Debian-Based Distributions

The following command should install the necessary packages:

sudo apt-get install xmltv libxml-simple-perl libjavascript-perl libalgorithm-diff-perl \
                     libgetopt-mixed-perl libcompress-zlib-perl libdata-dumper-simple-perl \
                     libdate-manip-perl liblist-compare-perl libdatetime-format-strptime-perl \
                     libhtml-parser-perl libxml-dom-perl libgd-gd2-perl libdigest-sha1-perl \
                     libarchive-zip-perl libio-string-perl

Fedora Core 6 (FC6)

Providing you have your yum/smart system setup for ATrpms.net (see http://www.atrpms.net/install.html), all of the required modules can be installed with:

sudo yum -y install perl-GD perl-Algorithm-Diff perl-XML-Parser perl-TermReadKey \
  js js-devel perl-XML-XQL perl-IO-String perl-XML-Simple perl-DateTime-Format-Builder \
  perl-List-Compare perl-DateTime-Format-Strptime tor xmltv

cd /tmp
wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.60.tar.gz
tar xvzf js-1.60.tar.gz
cd js/src
make -f Makefile.ref BUILD_OPT=1
cp jsopcode.tbl /usr/include

cpan JavaScript
cpan
cpan> force install JavaScript

Mythdora 4

The process is very similar to that of the Fedora Core 6:

su

yum -y install perl-GD perl-Algorithm-Diff perl-XML-Parser perl-TermReadKey \
  js js-devel perl-XML-XQL perl-IO-String perl-XML-Simple perl-DateTime-Format-Builder \
  perl-List-Compare perl-DateTime-Format-Strptime tor xmltv gcc

Digest::SHA1 does not come with Mythdora so you will need to install it with cpan. The first time you run cpan you will be asked to configure it, the default options should be fine.

cpan Digest::SHA1

cd /tmp
wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.60.tar.gz
tar xvzf js-1.60.tar.gz
cd js/src
make -f Makefile.ref BUILD_OPT=1
cp jsopcode.tbl /usr/include

cpan JavaScript
cpan
cpan> force install JavaScript

Gentoo

Some (but not all) of the mandatory modules can be installed with:

sudo emerge -av xmltv Algorithm-Diff Digest-SHA1

For the rest, you can use g-cpan (e.g. "sudo g-cpan -i List::Compare"). You may wish to useeix to search for new ebuilds before installing (e.g. "eix -s List::Compare").

Alternately, you can use the non-distribution specific method.

Non-Distribution Specific

The general method to install Perl modules is to do this (as root):

cpan <module>

For example:

cpan List::Compare

This will install the mandatory Perl modules:

cpan XMLTV::Ask Algorithm::Diff Compress::Zlib Cwd Data::Dumper Date::Manip Getopt::Long \
     List::Compare LWP::UserAgent POSIX Digest::SHA1

And this will install the optional ones:

cpan DateTime::Format::Strptime File::Basename File::Path GD HTML::Entities \
     HTML::TokeParser HTML::TreeBuilder IO::File Storable Time::HiRes XML::DOM \
     XML::DOM::NodeList XML::Simple Storable HTTP::Cookies File::Basename \
     LWP::ConnCache GD Digest::MD5 Archive::Zip IO::String \
     DateTime::Format::Strptime JavaScript

(Note: this list may not be up to date. If Shepherd complains about more missing modules, install those with cpan <module> as described above.)

It's safe to run these commands even if you have some modules installed; it will leave any previously-installed modules alone, so long as they're the most recent versions.

Note: XMLTV

XMLTV can also be downloaded from http://files.xmltv.org/.

If you are unsure about whether you have XMLTV installed, run the command perldoc XMLTV and see if there is any documentation on your system. If there is, then it is already installed.

XMLTV version 0.5.44 or later is recommended as it supports HDTV flags.

Note: JavaScript

The JavaScript module enables grabbers to read web pages with embedded Javascript. It can be a difficult module to install on some (non-Debian-based) distributions, but provides access to higher-quality TV guide data.

If you're running a RedHat Fedora system, you can verify whether you have the required Javascript perl module bindings js and js-devel through RPM:

# rpm -q js js-devel
js-1.5-2.fc3
js-devel-1.5-2.fc3

Providing rpm responds with something similar to above, the javascript library files are installed and you can proceed with installing the perl javascript bindings. If rpm indicates the packages are not installed, you will need to install them from a RPM repository. [rpmfind.net] can be useful here:

• http://rpmfind.net/linux/rpm2html/search.php?query=js-devel
• http://rpmfind.net/linux/rpm2html/search.php?query=js

The js and js-devel libraries can be installed on a Gentoo System by the spidermonkey ebuild, but a symbolic link must be created to put them in the location expected by Javascript:

emerge spidermonkey
mkdir /usr/lib/MozillaFirefox/
ln -s /usr/include/ /usr/lib/MozillaFirefox/

Once you have installed the javascript library files, you can install the Javascript perl bindings from CPAN:

cpan Javascript

Optional Software: Tor

Some grabbers work faster/better if they can operate using The Onion Router (tor) from http://tor.eff.org/. Once Tor is installed, shepherd will automatically find it and start using it. No configuration of Tor is necessary.

Tor is available in RPM form from as http://tor.eff.org/dist/rpm/tor-0.1.1.25-tor.0.rh4_4.i386.rpm and http://dag.wieers.com/packages/libevent/libevent-1.2-1.fc3.rf.i386.rpm

For Debian-based installations, a simple "sudo apt-get install tor" will get things working.

Security

Shepherd automatically checks for new versions of itself and its components, downloads, and executes them. See Security for details and implications.

Using Shepherd with multiple sources

It is possible to use a single installation of Shepherd to grab data for multiple sources (such as FTA & Foxtel in the following example).

You can do this by using the following steps:

1. Using mythtv-setup create two channel sources, one for FTA and another for Foxtel.
2. Select tv_grab_au as the grabber for both of these sources
3. Add cards and channels to the appropriate sources as required
4. Configure shepherd to grab data for both the FTA and Foxtel channels that you have configured in Myth
5. From a frontend, select setup -> general. On the last page you can schedule mythfilldatabase to be run automatically. This page also allows you to add parameters to the mythfilldatabase command line, to which you must add '--update'. This prevents Myth from adding the Foxtel channels to the FTA source, and vice-versa.

Whilst Mythfilldatabase will call Shepherd twice (once for each source), data will only be downloaded once for both FTA and Foxtel during the first run, as Shepherd will automatically use cached data when run twice in a short period of time.

Troubleshooting

For further information, please see the FAQ.

JavaScript with Suse 10.1, 10.2 and 10.3

After installing libjs and libjs-devel you will still be missing some files to get 'cpan JavaScript' working please be careful with the following it is more of a hack than a fix.

wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.60.tar.gz
tar zxvf js-1.60.tar.gz
cd js/src
make -f Makefile.ref BUILD_OPT=1
cp js*.h /usr/include/
cp jsopcode.tbl /usr/include/
cd Linux_All_OPT.OBJ
cp js*.h /usr/include/
cp libjs.* /usr/include/
ldconfig

‘cpan JavaScript’ will still have issues but should be about 86% successful, you can now do a force install by

pc:/ # cpan
cpan> force install JavaScript

If the compile complains about missing include files, but they exist on your system in a different directory tree, maybe link the trees: [in my case, one of the directories in the "-I" include-file search path was a non-existant /usr/lib/MozillaFirefox, but they exist under /usr/lib/mozilla-firefox]

cd /usr/lib
ln -s mozilla-firefox MozillaFirefox

This will cause the files to be found under both the 'real' path, and the path that JavaScript thinks they should be, and a re-attempt to install JavaScript worked OK.