Wiki Navigation

Installation

It's not important which user you install and run Shepherd as, so long as you're consistent. We do, however, caution against running Shepherd as root.

[MythTV users only] We no longer recommend setting up MythTV to automatically run Shepherd due to the potentially confusing steps required. (In particular, it can be hard to know which user your system will run it as.) For simplicity, we instead describe how to set up Shepherd to supply MythTV with data via a cron job.

Configure MythTV first. In mythtv-setup -> 3. Video Sources -> (your video source) -> Listings grabber, select No grabber. Perform a channel scan: mythtv-setup -> 5. Channel Editor -> Channel Scanner.

Download Shepherd:

wget http://www.whuffy.com/shepherd/shepherd

Run it:
```
perl shepherd
```
If Shepherd won't run because a "mandatory module" is not found, install it (see "Perl Dependencies") and try again. You may have to install several mandatory Perl modules before Shepherd can run.

Shepherd will now install itself and its components (probably into ~/.shepherd/). Most likely, one or more components will fail due to missing Perl modules, but that's fine: we'll fix that later.
Answer the configuration questions to select your region. When asked if you want Guided Channel Selection, say "yes". Shepherd will then step through each of your MythTV channels, asking you to choose appropriate guide data for each one. Some MythTV channels you won't want guide data for (e.g. radio channels, guide channels, duplicate channels); that's fine.

If you are not a MythTV user, or Shepherd cannot access your MythTV, you will need to use Advanced Channel Selection instead to manually enter XMLTV IDs for each channel.

When asked to confirm that you want to create Shepherd's configuration file, say yes.
Shepherd will test its components (some of which may fail) and ask if you would like to install channel icons now. This is optional, and you can do it later if you prefer. If you say yes, Shepherd will step you through a variety of icon galleries to choose from.
Shepherd will ask if you want to grab data now. Say no. Shepherd will exit.
For maximum functionality, install any missing Perl modules required by components. How to do this varies depending on your distribution: see the detailed instructions and why this is worth doing. Essentially, you should attempt to install any modules that Shepherd complains are missing when you do this:
```
~/.shepherd/shepherd --check
```
To avoid confusion, delete the original Shepherd file you downloaded (leaving the newly installed version in ~/.shepherd/).
[optional] Install Tor if you wish to increase the potential speed of some grabbers. (Currently, no grabbers use Tor, so as of writing there is no benefit to this step.)

Shepherd is now installed. This means that when run, it will create a file of TV guide data for you (by default: ~/.shepherd/output.xmltv). It also creates a log file at ~/.shepherd/log/shepherd.log.

Most users want this guide data to be regularly fed to another program: to set this up, see the relevant section below on integration with MythTV, Freevo, or EyeTV.

Integration with MythTV

Cron Job

As mentioned above, it's simpler to avoid MythTV's automated process for collecting guide data, and instead use a cron job. Enter:

crontab -e

... and add the following line:

44 * * * * ~/.shepherd/shepherd --daily --quiet && mythfilldatabase --file 1 ~/.shepherd/output.xmltv --quiet

The "44 * * * *" part means it'll run at the 44th minute of every hour. (If convenient, please pick a random number between 00 and 59, rather than 44, so not everybody runs at the same time.) The "--daily" option means Shepherd will only attempt to grab data if it's been about a day since it last did so successfully. The program mythfilldatabase is invoked to read Shepherd's output into MythTV, and the --quiet options prevent you from having output emailed to you.

Once you've added that line and saved and exited, you should see:

$ crontab -e
crontab: installing new crontab

At 44 minutes past the hour (or at whatever time you chose), Shepherd will begin gathering guide data, then feed it to MythTV via mythfilldatabase.

Multiple sources

If you use multiple sources (e.g. free to air plus cable TV), your cron job should instead look like this:

44 * * * * ~/.shepherd/shepherd --daily --quiet && mythfilldatabase --update --file 1 ~/.shepherd/output.xmltv --quiet && mythfilldatabase --update --file 2 ~/.shepherd/output.xmltv --quiet

The differences are the "--update" option, which prevents MythTV from creating channels on the wrong source, and that mythfilldatabase is invoked twice, once for each source.

The old method

The alternative to a cron job is to have MythTV automatically run Shepherd. This involves creating a symlink to Shepherd from /usr/bin/tv_grab_au and selecting "Australia (tv_grab_au) in mythtv-setup -> Video Sources. You also have to run the MythTV frontend, and in Utilities/Setup -> Setup -> General -> Mythfilldatabase, check "Automatically run mythfilldatabase".

The main problem with this method is that MythTV may not invoke Shepherd as the user you expect: it might be your own user, it might be mythtv, or it might be root, depending on your system. This in turn can lead to Bad Things, such as root creating files that your regular Shepherd user cannot modify.

Integration with Freevo

Freevo will use the Shepherd data with a little alteration. In local_conf.py set the XMLTV_GRABBER XMLTV_GRABBER = '/home/tv/.shepherd/tv_grab_au --timeoffset=Auto' (or /usr/bin/tv_grab_au if you have set the symlink) If you have existing xmltv channel id set, use the ones you have already set when running

~/.shepherd/shepherd --configure

Otherwise use any channel id you choose. Test your setup with

freevo tv_grab

Integration with EyeTV

Beginning with EyeTV version 3.0.1, EyeTV can use xmltv files as its source for channel and program guide data. However, it is quite picky about how those files must be presented, so it won't quite work "out-of-the-box" with Shepherd. To integrate Shepherd with EyeTV, you need to do the following:

install and configure Shepherd
install and launch EyeTV
in EyeTV's Preferences, select "None" as the TV guide service
download the XMLTV DTD and save it (using the file name "xmltv.dtd") in your .shepherd directory
```
cd ~/.shepherd
wget http://xmltv.cvs.sourceforge.net/*checkout*/xmltv/xmltv/xmltv.dtd
```
EyeTV won't recognise the default output file suffix of ".xmltv", so configure Shepherd to produce an output file with the suffix ".xml", e.g.
```
./tv_grab_au --component-set shepherd:output=output.xml
```
now run (or re-run) Shepherd
```
./tv_grab_au
```
open the resulting .xml file with EyeTV, either by dragging and dropping in the finder or using the following command line:
```
open -a EyeTV.app output.xml
```
in EyeTV, set up each channel in your list to use the xmltv guide information (in the Channels list, select xmltv, and choose the matching channel name from the list)
you should now see your download xmltv information when you view the Program Guide or your favourite channels list(s) in EyeTV. xmltv information will also be used by any Smart Guides you have set up.

To periodically run Shepherd, set up a sequence of cron jobs, such as the following (these are spaced one hour apart to give Shepherd plenty of time to finish):

01 03 * * * /Users/yourname/.shepherd/tv_grab_au > /dev/null
01 04 * * * open -a EyeTV.app /Users/yourname/.shepherd/output.xml

Another possibility would be to use AppleScript?, but that probably makes life harder rather than easier.

Some Useful Commands

A full list of command-line options:

~/.shepherd/shepherd --help

The status screen:

~/.shepherd/shepherd --status

What Shepherd's been up to:

~/.shepherd/shepherd --history

Check your channel mappings:

~/.shepherd/shepherd --show-channels

Re-configure Shepherd (preserving old settings except where overridden):

~/.shepherd/shepherd --configure

Check whether any components are missing Perl modules or need to be configured:

~/.shepherd/shepherd --check

Check your channel mappings:

~/.shepherd/shepherd --show-channels

Perl Dependencies

Shepherd and its components rely on various third-party Perl modules. Some are mandatory, meaning that Shepherd won't run without them. Most, however, are optional, meaning that Shepherd can still operate without them, but some of its components (including grabbers) won't.

You should install all the optional modules if possible, as this increases Shepherd's efficiency, reliability, and possibly the quality of the guide data it delivers. For more information, see the FAQ.

To see which Perl modules you're missing, do this:

~/.shepherd/shepherd --check

If applicable, follow the distribution-specific instructions for Debian-based, Fedora Core 6, and Gentoo. Alternately you can use the general method.

Debian-Based Distributions (including Ubuntu and its derivatives)

The following command should install the necessary packages:

sudo apt-get install xmltv libxml-simple-perl libjavascript-perl libalgorithm-diff-perl \
                     libgetopt-mixed-perl libcompress-zlib-perl libdata-dumper-simple-perl \
                     libdate-manip-perl liblist-compare-perl libdatetime-format-strptime-perl \
                     libhtml-parser-perl libxml-dom-perl libgd-gd2-perl libdigest-sha1-perl \
                     libarchive-zip-perl libio-string-perl libdbi-perl

Fedora Core 6 (FC6)

Providing you have your yum/smart system setup for ATrpms.net (see http://www.atrpms.net/install.html), all of the required modules can be installed with:

An older version of the perl JavaScript module needs to be installed since the newer versions (after 1.04) do not install properly even with the force option. Just take the default answers to the three questions about SpiderMonkey?

sudo yum -y install perl-GD perl-Algorithm-Diff perl-XML-Parser perl-TermReadKey \
  js js-devel perl-XML-XQL perl-IO-String perl-XML-Simple perl-DateTime-Format-Builder \
  perl-List-Compare perl-DateTime-Format-Strptime tor xmltv

cd /tmp
wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.60.tar.gz
tar xvzf js-1.60.tar.gz
cd js/src
make -f Makefile.ref BUILD_OPT=1
cp jsopcode.tbl /usr/include

cpan -f C/CL/CLAESJAC/JavaScript-1.04.tar.gz

Mythdora 4 and 5

The process is very similar to that of the Fedora Core 6:

su

yum -y install perl-GD perl-Algorithm-Diff perl-XML-Parser perl-TermReadKey \
  js js-devel perl-XML-XQL perl-IO-String perl-XML-Simple perl-DateTime-Format-Builder \
  perl-List-Compare perl-DateTime-Format-Strptime tor xmltv gcc

Digest::SHA1 does not come with Mythdora 4 so you will need to install it with cpan. The first time you run cpan you will be asked to configure it, the default options should be fine.

cpan Digest::SHA1

On Mythdora 5 Digest::SHA1 can be installed as an rpm.

yum install perl-Digest-SHA1 -y

cd /tmp
wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.60.tar.gz
tar xvzf js-1.60.tar.gz
cd js/src
make -f Makefile.ref BUILD_OPT=1
cp jsopcode.tbl /usr/include

cpan -f C/CL/CLAESJAC/JavaScript-1.04.tar.gz

Gentoo

Some (but not all) of the mandatory modules can be installed with:

sudo emerge -av xmltv Algorithm-Diff Digest-SHA1

For the rest, you can use g-cpan (e.g. "sudo g-cpan -i List::Compare"). You may wish to useeix to search for new ebuilds before installing (e.g. "eix -s List::Compare").

Alternately, you can use the non-distribution specific method.

Rex currently (Aug_08) requires Apache2::Filter which won't compile from g-cpan. Installing mod_perl has many dependencies but will allow the grabber to run.

Non-Distribution Specific

The general method to install Perl modules is to do this (as root):

cpan <module>

For example:

cpan List::Compare

This will install the mandatory Perl modules:

cpan XMLTV::Ask Algorithm::Diff Compress::Zlib Cwd Data::Dumper Date::Manip Getopt::Long \
     List::Compare LWP::UserAgent POSIX Digest::SHA1

And this will install the optional ones:

cpan DateTime::Format::Strptime File::Basename File::Path GD HTML::Entities \
     HTML::TokeParser HTML::TreeBuilder IO::File Storable Time::HiRes XML::DOM \
     XML::DOM::NodeList XML::Simple Storable HTTP::Cookies File::Basename \
     LWP::ConnCache GD Digest::MD5 Archive::Zip IO::String \
     DateTime::Format::Strptime JavaScript

(Note: this list may not be up to date. If Shepherd complains about more missing modules, install those with cpan <module> as described above.)

It's safe to run these commands even if you have some modules installed; it will leave any previously-installed modules alone, so long as they're the most recent versions.

Note: XMLTV

XMLTV can also be downloaded from http://files.xmltv.org/.

If you are unsure about whether you have XMLTV installed, run the command perldoc XMLTV and see if there is any documentation on your system. If there is, then it is already installed.

XMLTV version 0.5.44 or later is recommended as it supports HDTV flags.

Note: JavaScript

The JavaScript module enables grabbers to read web pages with embedded Javascript. It can be a difficult module to install on some (non-Debian-based) distributions, but provides access to higher-quality TV guide data.

If you're running a RedHat Fedora system, you can verify whether you have the required Javascript perl module bindings js and js-devel through RPM:

# rpm -q js js-devel
js-1.5-2.fc3
js-devel-1.5-2.fc3

Providing rpm responds with something similar to above, the javascript library files are installed and you can proceed with installing the perl javascript bindings. If rpm indicates the packages are not installed, you will need to install them from a RPM repository. [rpmfind.net] can be useful here:

The js and js-devel libraries can be installed on a Gentoo System by the spidermonkey ebuild, but a symbolic link must be created to put them in the location expected by Javascript:

emerge spidermonkey
mkdir /usr/lib/MozillaFirefox/
ln -s /usr/include/ /usr/lib/MozillaFirefox/

Once you have installed the javascript library files, you can install the Javascript perl bindings from CPAN:

cpan JavaScript

Optional Software: Tor

Some grabbers work faster/better if they can operate using The Onion Router (tor) from http://tor.eff.org/. Once Tor is installed, shepherd will automatically find it and start using it. No configuration of Tor is necessary.

Tor is available in RPM form from as http://tor.eff.org/dist/rpm/tor-0.1.1.25-tor.0.rh4_4.i386.rpm and http://dag.wieers.com/packages/libevent/libevent-1.2-1.fc3.rf.i386.rpm

For Debian-based installations, a simple "sudo apt-get install tor" will get things working.

Security

Shepherd automatically checks for new versions of itself and its components, downloads, and executes them. See Security for details and implications.

Using Shepherd with multiple sources

It is possible to use a single installation of Shepherd to grab data for multiple sources (such as FTA & Foxtel in the following example).

You can do this by using the following steps:

Using mythtv-setup create two channel sources, one for FTA and another for Foxtel.
Select tv_grab_au as the grabber for both of these sources
Add cards and channels to the appropriate sources as required
Configure shepherd to grab data for both the FTA and Foxtel channels that you have configured in Myth
From a frontend, select setup -> general. On the last page you can schedule mythfilldatabase to be run automatically. This page also allows you to add parameters to the mythfilldatabase command line, to which you must add '--update'. This prevents Myth from adding the Foxtel channels to the FTA source, and vice-versa.

Whilst Mythfilldatabase will call Shepherd twice (once for each source), data will only be downloaded once for both FTA and Foxtel during the first run, as Shepherd will automatically use cached data when run twice in a short period of time.

Troubleshooting

For further information, please see the FAQ.

JavaScript with Suse 10.1, 10.2 and 10.3

After installing libjs and libjs-devel you will still be missing some files to get 'cpan !JavaScript' working please be careful with the following it is more of a hack than a fix.

wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.60.tar.gz
tar zxvf js-1.60.tar.gz
cd js/src
make -f Makefile.ref BUILD_OPT=1
cp js*.h /usr/include/
cp jsopcode.tbl /usr/include/
cd Linux_All_OPT.OBJ
cp js*.h /usr/include/
cp libjs.* /usr/include/
ldconfig

‘cpan JavaScript’ will still have issues but should be about 86% successful, you can now do a force install by

pc:/ # cpan
cpan> force install JavaScript

If the compile complains about missing include files, but they exist on your system in a different directory tree, maybe link the trees: [in my case, one of the directories in the "-I" include-file search path was a non-existant /usr/lib/MozillaFirefox, but they exist under /usr/lib/mozilla-firefox]

cd /usr/lib
ln -s mozilla-firefox MozillaFirefox

This will cause the files to be found under both the 'real' path, and the path that JavaScript thinks they should be, and a re-attempt to install JavaScript worked OK.

Download in other formats:

Plain Text