Man Linux: Main Page and Category List

NAME

       htdig - retrieve HTML documents for ht://Dig search engine

SYNOPSIS

       htdig [options]

DESCRIPTION

       Htdig  retrieves  HTML  documents  using  the HTTP protocol and gathers
       information from these documents which can  later  be  used  to  search
       these documents. This program can be referred to as the search robot.

OPTIONS

       -      Get the list of URLs to start indexing from standard input. This
              will override the default parameter start_url specified  in  the
              config file  and the file supplied to the -m option.

       -a     Use  alternate  work  files.  Tells  htdig  to  append  .work to
              database files, causing a second copy  of  the  database  to  be
              built.  This  allows  the  original files to be used by htsearch
              during the indexing run.

       -c configfile
              Use the specified configfile instead of the default.

       -h maxhops
              Restrict the dig to documents that are  at  most  maxhops  links
              away from the starting document. This only works if option -i is
              also given.

       -i     Initial. Do not use any old databases.  Old  databases  will  be
              erased before running the program.

       -m filename
              Minimal  run.  Only  index  the URLs given in the file filename,
              ignoring all others. URLs in the file should  be  formatted  one
              URL per line.

       -s     Print statistics about the dig after completion.

       -t     Create  an ASCII version of the document database. This database
              is easy to parse with other programs so that information can  be
              extracted  from  it for purposes other than searching. One could
              gather some interesting statistics from this database.

              Fieldname                           Value
                  u       URL
                  t       Title
                  a       State
                          (0 normal, 1 not found, 2 not indexed, 3 obsolete)
                  m       Time of last modification reported by the server
                  s       Document Size in bytes
                  H       Excerpt of the document
                  h       Meta Description
                  l       Time of last retrieval
                  L       Count of links in the document or  of outgoing links
                  b       Number of links to the document, also called
                          incoming links or backlinks
                  c       Hop count of this document
                  g       Signature of this document
                          (used to detect duplicates)
                  e       E-Mail address to use for a notification from htnotify

                  n       Date on which such notification is sent
                  S       Subject of the notifcation message
                  d       The text of Incoming links pointing to this document
                          (e.g. <a href="docURL">description</a>)
                  A       Anchors in the document (i.e. <A NAME=...)

       -u username:password
              Tells htdig to send the supplied username and password with each
              HTTP  request. The credentials will be encoded using the Basic
              authentication method. There HAS to be a colon (:)  between  the
              username and password.

       -v     Verbose mode. This increases the verbosity of the program. Using
              more than 2 is probably only useful for debugging purposes.  The
              default  verbose  mode (using only one -v) gives a nice progress
              report while digging. Please consult the section  below  on  the
              exact format of the progress report.

       FORMAT OF THE PROGRESS REPORT GIVEN IN VERBOSE MODE
              A  line is shown for each URL, with 3 numbers before the URL and
              some symbols after the URL. The first number is  the  number  of
              documents  parsed  so  far,  the  second  is  the DocID for this
              document, and the third is the hop count of the document (number
              of  hops  from one of the start_url documents). Signification of
              the symbols printed after the url:

              "*" is printed for a link already visited

              "+" is printed for a new link just queued

              "-" is output for a  link  rejected  for  any  of  a  number  of
              reasons.  To  find  out  what those reasons are, you need to run
              htdig with at least 3 -v options, i.e. -vvv.

       If there are no "*", "+" or "-" symbols after the URL, it doesn’t  mean
       the  document  was  not  parsed or was empty, but only that no links to
       other documents were found within it. With more verbose  output,  these
       symbols will get interspersed in several lines of debugging output.

       FILES

       /etc/htdig/htdig.conf
              The default configuration file.

SEE ALSO

       Please  refer  to  the  HTML   pages   (in   the   htdig-doc   package)
       /usr/share/doc/htdig-doc/html/index.html    and    the   manual   pages
       htdigconfig(8) , htmerge(1) , htnotify(1) , htsearch(1)  and  rundig(1)
       for a detailed description of ht://Dig and its commands.

AUTHOR

       This manual page was written by Christian Schwarz, modified by Stijn de
       Bekker. It is updated and maintained by Robert Ribnitz and based on the
       HTML documentation of ht://Dig.

                                 21 July 1997                         htdig(1)