libextractor

GNU libextractor
Log | Files | Refs | Submodules | README | LICENSE

libextractor.texi (46287B)


      1 \input texinfo                  @c -*- Texinfo -*-
      2 @c % The structure of this document is based on the
      3 @c % Texinfo manual from libgcrypt by Werner Koch and 
      4 @c % and Moritz Schulte.
      5 @c %**start of header
      6 @setfilename libextractor.info
      7 @include version.texi
      8 @settitle The GNU libextractor Reference Manual
      9 @c Unify all the indices into concept index.
     10 @syncodeindex fn cp
     11 @syncodeindex vr cp
     12 @syncodeindex ky cp
     13 @syncodeindex pg cp
     14 @syncodeindex tp cp
     15 @c %**end of header
     16 @copying
     17 This manual is for GNU libextractor
     18 (version @value{VERSION}, @value{UPDATED}), a library for metadata
     19 extraction.
     20 
     21 Copyright @copyright{} 2007, 2010, 2012 Christian Grothoff
     22 
     23 @quotation
     24 Permission is granted to copy, distribute and/or modify this document
     25 under the terms of the GNU Free Documentation License, Version 1.3
     26 or any later version published by the Free Software Foundation;
     27 with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
     28 Texts.  A copy of the license is included in the section entitled ``GNU
     29 Free Documentation License''.
     30 @end quotation
     31 @end copying
     32 
     33 @dircategory Software libraries
     34 @direntry
     35 * Libextractor: (libextractor).    Metadata extraction library.
     36 @end direntry
     37 
     38 
     39 
     40 @c
     41 @c Titlepage
     42 @c
     43 @titlepage
     44 @title The GNU libextractor Reference Manual
     45 @subtitle Version @value{VERSION}
     46 @subtitle @value{UPDATED}
     47 @author Christian Grothoff (@email{christian@@grothoff.org})
     48 
     49 @page
     50 @vskip 0pt plus 1filll
     51 @insertcopying
     52 @end titlepage
     53 
     54 @summarycontents
     55 @contents
     56 
     57 
     58 @ifnottex
     59 @node Top
     60 @top The GNU libextractor Reference Manual
     61 @insertcopying
     62 @end ifnottex
     63 
     64 @menu
     65 * Introduction::                 What is GNU libextractor.
     66 * Preparation::                  What you should do before using the library.
     67 * Generalities::                 General library functions and data types.
     68 * Extracting meta data::         How to use GNU libextractor to obtain meta data.
     69 * Language bindings::            How to use GNU libextractor from languages other than C.
     70 * Utility functions::            Utility functions of GNU libextractor.
     71 * Existing Plugins::             What plugins are available.
     72 * Writing new Plugins::          How to write new plugins for GNU libextractor.
     73 * Internal utility functions::   Utility functions of GNU libextractor for writing plugins.
     74 * Reporting bugs::               How to report bugs or request new features.
     75 
     76 Appendices
     77 
     78 * GNU Free Documentation License::  Copying this manual.
     79 
     80 Indices
     81 
     82 * Index::                       Index
     83 @c * Function and Data Index::     Index of functions, variables and data types.
     84 @c * Type Index::                  Index of data types.
     85 
     86 @end menu
     87 
     88 
     89 
     90 @c **********************************************************
     91 @c *******************  Introduction  ***********************
     92 @c **********************************************************
     93 @node Introduction
     94 @chapter Introduction
     95 
     96 @cindex error handling
     97 GNU libextractor is GNU's library for extracting meta data from
     98 files.  Meta data includes format information (such as mime type,
     99 image dimensions, color depth, recording frequency), content
    100 descriptions (such as document title or document description) and
    101 copyright information (such as license, author and contributors).
    102 Meta data extraction is an inherently uncertain business --- a parse
    103 error can be a corrupt file, an incompatibility in the file format
    104 version, an entirely different file format or a bug in the parser.  As
    105 a result of this uncertainty, GNU libextractor deliberately
    106 avoids to ever report any errors.  Unexpected file contents simply
    107 result in less or possibly no meta data being extracted.  
    108 
    109 @cindex plugin
    110 GNU libextractor uses plugins to handle various file formats.
    111 Technically a plugin can support multiple file formats; however, most
    112 plugins only support one particular format.  By default,
    113 GNU libextractor will use all plugins that are available and found
    114 in the plugin installation directory.  Applications can
    115 request the use of only specific plugins or the exclusion of
    116 certain plugins.
    117 
    118 GNU libextractor is distributed with the @command{extract} 
    119 command@footnote{Some distributions ship @command{extract} in a
    120 seperate package.} which is a command-line tool for extracting
    121 meta data.  @command{extract} is given a list of filenames and 
    122 prints the resulting meta data to the console.  The @command{extract}
    123 source code also serves as an advanced example for how to use
    124 GNU libextractor.  
    125 
    126 This manual focuses on providing documentation for writing software
    127 with GNU libextractor.  The only relevant parts for end-users
    128 are the chapter on compiling and installing GNU libextractor
    129 (@xref{Preparation}.).  Also, the chapter on existing plugins maybe of
    130 interest (@xref{Existing Plugins}.).  Additional documentation for
    131 end-users can be find in the man page on @command{extract} (using
    132 @verb{|man extract|}).
    133 
    134 @cindex license
    135 GNU libextractor is licensed under the GNU General Public License,
    136 specifically, since version 0.7, GNU libextractor is licensed under GPLv3
    137 @emph{or any later version}.
    138 
    139 @node Preparation
    140 @chapter Preparation
    141 
    142 This chapter first describes the general build instructions that
    143 should apply to all systems.  Specific instructions for known problems
    144 for particular platforms are then described in individual sections
    145 afterwards.
    146 
    147 Compiling GNU libextractor follows the standard GNU autotools build process
    148 using @command{configure} and @command{make}.  For details on the GNU
    149 autotools build process, read the @file{INSTALL} file and query
    150 @verb{|./configure --help|} for additional options.  
    151 
    152 GNU libextractor has various dependencies, most of which are optional. 
    153 Instead of specifying the names of the software packages, we
    154 will give the list in terms of the names of the respective
    155 Debian (wheezy) packages that should be installed.
    156 
    157 You absolutely need:
    158 
    159 @itemize @bullet
    160 @item
    161 libtool
    162 @item
    163 gcc
    164 @item
    165 make
    166 @item
    167 g++ 
    168 @item
    169 libltdl7-dev
    170 @end itemize
    171 
    172 Recommended dependencies are:
    173 @itemize @bullet
    174 @item
    175 zlib1g-dev
    176 @item
    177 libbz2-dev
    178 @item
    179 libgif-dev
    180 @item
    181 libvorbis-dev
    182 @item
    183 libflac-dev
    184 @item
    185 libmpeg2-4-dev
    186 @item
    187 librpm-dev
    188 @item
    189 libgtk2.0-dev or libgtk3.0-dev
    190 @item
    191 libgsf-1-dev
    192 @item
    193 libqt4-dev
    194 @item
    195 libpoppler-dev
    196 @item
    197 libexiv2-dev
    198 @item
    199 libavformat-dev
    200 @item
    201 libswscale-dev
    202 @item
    203 libswresample-dev
    204 @item
    205 libgstreamer1.0-dev
    206 @item
    207 libgstreamer-plugins-base1.0-dev
    208 @end itemize
    209 
    210 For Subversion access and compilation one also needs:
    211 @itemize @bullet
    212 @item
    213 subversion
    214 @item
    215 autoconf
    216 @item
    217 automake
    218 @end itemize
    219 
    220 Please notify us if we missed some dependencies (note that the list is
    221 supposed to only list direct dependencies, not transitive
    222 dependencies).
    223 
    224 Once you have compiled and installed GNU libextractor, you should have a file
    225 @file{extractor.h} installed in your @file{include/} directory.  This
    226 file should be the starting point for your C and C++ development with
    227 GNU libextractor.  The build process also installs the @file{extract} binary and
    228 man pages for @file{extract} and GNU libextractor.  The @file{extract} man page
    229 documents the @file{extract} tool.  The GNU libextractor man page gives a brief
    230 summary of the C API for GNU libextractor.
    231 
    232 @cindex packageing
    233 @cindex directory structure
    234 @cindex plugin
    235 @cindex environment variables
    236 @vindex LIBEXTRACTOR_PREFIX
    237 When you install GNU libextractor, various plugins will be
    238 installed in the @file{lib/libextractor/} directory.  The main library
    239 will be installed as @file{lib/libextractor.so}.  Note that
    240 GNU libextractor will attempt to find the plugins relative to the
    241 path of the main library.  Consequently, a package manager can move
    242 the library and its plugins to a different location later --- as long
    243 as the relative path between the main library and the plugins is
    244 preserved.  As a method of last resort, the user can specify an
    245 environment variable @verb{|LIBEXTRACTOR_PREFIX|}.  If
    246 GNU libextractor cannot locate a plugin, it will look in
    247 @verb{|LIBEXTRACTOR_PREFIX/lib/libextractor/|}.
    248 
    249 
    250 @section Installation on GNU/Linux
    251 
    252 Should work using the standard instructions without problems.
    253 
    254 
    255 @section Installation on FreeBSD
    256 
    257 Should work using the standard instructions without problems.
    258 
    259 
    260 @section Installation on OpenBSD
    261 
    262 OpenBSD 3.8 also doesn't have CODESET in @file{langinfo.h}.  CODESET
    263 is used in GNU libextractor in about three places.  This causes problems
    264 during compilation.
    265 
    266 
    267 @section Installation on NetBSD
    268 
    269 No reports so far.
    270 
    271 
    272 @section Installation using MinGW
    273 
    274 Linking -lstdc++ with the provided libtool fails on Cygwin, this
    275 is a problem with libtool, there is unfortunately no flag to tell
    276 libtool how to do its job on Cygwin and it seems that it cannot be the
    277 default to set the library check to 'pass_all'.  Patching libtool may
    278 help.
    279 
    280 Note: this is a rather dated report and may no longer apply.
    281 
    282 
    283 @section Installation on OS X
    284 
    285 libextractor has two installation methods on Mac OS X: it can be
    286 installed as a Mac OS X framework or with the standard
    287 @command{./configure; make; make install} shell commands. The
    288 framework package is self-contained, but currently omits some of the
    289 extractor plugins that can be compiled in if libextractor is installed
    290 with @command{./configure; make; make install} (provided that the
    291 required dependencies exist.)
    292 
    293 @subsection Installing and uninstalling the framework
    294 
    295 The binary framework is distributed as a disk image (@file{Extractor-x.x.xx.dmg}).
    296 Installation is done by opening the disk image and clicking @file{Extractor.pkg}
    297 inside it. The Mac OS X installer application will then run. The framework
    298 is installed to the root volume's @file{/Library/Frameworks} folder and installing
    299 will require admin privileges.
    300 
    301 The framework can be uninstalled by dragging @*
    302 @file{/Library/Frameworks/Extractor.framework} to the @file{Trash}.
    303 
    304 
    305 @subsection Using the framework
    306 
    307 In the framework, the @command{extract} command line tool can be found at @*
    308 @file{/Library/Frameworks/Extractor.framework/Versions/Current/bin/extract}
    309 
    310 The framework can be used in software projects as a framework or as a dynamic
    311 library. 
    312 
    313 When using the framework as a dynamic library in projects using autotools,
    314 one would most likely want to add  @*
    315 "-I/Library/Frameworks/Extractor.framework/Versions/Current/include"
    316 to CPPFLAGS and @*
    317 "-L/Library/Frameworks/Extractor.framework/Versions/Current/lib"
    318 to LDFLAGS.
    319 
    320 
    321 @subsection Example for using the framework
    322 
    323 @example
    324 @verbatim
    325 // hello.c
    326 #include <Extractor/extractor.h>
    327 
    328 int
    329 main (int argc, char **argv)
    330 {
    331   struct EXTRACTOR_PluginList *el;
    332   el = EXTRACTOR_plugin_load_defaults (EXTRACTOR_OPTION_DEFAULT_POLICY);
    333   // ...
    334   EXTRACTOR_plugin_remove_all (el);
    335   return 0;
    336 }
    337 @end verbatim
    338 @end example
    339 
    340 You can then compile the example using
    341 
    342 @verbatim
    343 $ gcc -o hello hello.c -framework Extractor
    344 @end verbatim
    345 
    346 @subsection Example for using the dynamic library
    347 
    348 @example
    349 @verbatim
    350 // hello.c
    351 #include <extractor.h>
    352 int main()
    353 {
    354   struct EXTRACTOR_PluginList *el;
    355   el = EXTRACTOR_plugin_load_defaults (EXTRACTOR_OPTION_DEFAULT_POLICY);
    356   // ...
    357   EXTRACTOR_plugin_remove_all (el);
    358   return 0;
    359 }
    360 @end verbatim
    361 @end example
    362 
    363 You can then compile the example using
    364 
    365 @verbatim
    366 $ gcc -I/Library/Frameworks/Extractor.framework/Versions/Current/include \
    367   -o hello hello.c \
    368   -L/Library/Frameworks/Extractor.framework/Versions/Current/lib \
    369   -lextractor
    370 @end verbatim
    371 
    372 Notice the difference in the @code{#include} line.
    373 
    374 
    375 
    376 
    377 
    378 
    379 @section Note to package maintainers
    380 
    381 The suggested way to package GNU libextractor is to split it into
    382 roughly the following binary packages:
    383 
    384 @itemize @bullet
    385 @item
    386 libextractor (main library only, only hard dependency for other packages depending on GNU libextractor)
    387 @item
    388 extract (command-line tool and man page extract.1)
    389 @item
    390 libextractor-dev (extractor.h header and man page libextractor.3)
    391 @item
    392 libextractor-doc (this manual)
    393 @item
    394 libextractor-plugins (plugins without external dependencies; recommended but not required by extract and libextractor package)
    395 @item
    396 libextractor-plugin-XXX (plugin with dependency on libXXX, for example for XXX=mpeg this would be @file{libextractor_mpeg.so})
    397 @item
    398 libextractor-plugins-all (meta package that requires all plugins except experimental plugins)
    399 @end itemize
    400 
    401 This would enable minimal installations (i.e. for embedded systems) to
    402 not include any plugins, as well as moderate-size installations (that
    403 do not trigger GTK and X11) for systems that have limited resources.
    404 Right now, the MP4 plugin is experimental and does nothing and should
    405 thus never be included at all; QuickTime, MP4, M4A and 3GPP files are
    406 instead handled by the @file{libextractor_qt.so} plugin, which only
    407 depends on zlib and is part of libextractor-plugins.  The gstreamer plugin is experimental
    408 but largely works with the correct version of gstreamer and can thus
    409 be packaged (especially if the dependency is available on the target
    410 system) but should probably not be part of libextractor-plugins-all.
    411 
    412 
    413 @node Generalities
    414 @chapter Generalities
    415 
    416 @section Introduction to the ``extract'' command
    417 
    418 The @command{extract} command takes a list of file names as arguments,
    419 extracts meta data from each of those files and prints the result to
    420 the console.  By default, @command{extract} will use all available
    421 plugins and print all (non-binary) meta data that is found.
    422 
    423 The set of plugins used by @command{extract} can be controlled using
    424 the ``-l'' and ``-n'' options.  Use ``-n'' to not load all of the
    425 default plugins.  Use ``-l NAME'' to specifically load a certain
    426 plugin.  For example, specify ``-n -l mime'' to only use the MIME
    427 plugin.
    428 
    429 The argument of ``-l'' is a colon-separated list of plugin names, each
    430 optionally preceded by a minus sign to remove the plugin instead of
    431 adding it and optionally followed by a plugin configuration in
    432 parentheses, as in ``-l "-pdf:pdf(fulltext)"''.  Only the last ``-l''
    433 option on the command line is used, so several plugins have to be
    434 named in a single list.  The options understood by the individual
    435 plugins are described in the chapter @emph{Existing Plugins}, under
    436 @emph{Video thumbnails and audio previews} and @emph{Full text}.
    437 
    438 Using the ``-p'' option the output of @command{extract} can be limited
    439 to only certain keyword types.  Similarly, using the ``-x'' option,
    440 certain keyword types can be excluded.  A list of all known keyword
    441 types can be obtained using the ``-L'' option.
    442 
    443 The output format of @command{extract} can be influenced with the
    444 ``-V'' (more verbose, lists filenames), ``-g'' (grep-friendly, all
    445 meta data on a single line per file) and ``-b'' (bibTeX style)
    446 options.
    447 
    448 @section Common usage examples for ``extract''
    449 
    450 The sample files used here are those of the test suite, in
    451 @file{src/plugins/testdata/}.  Note that the argument of @code{-l} is
    452 the short name of the plugin (@code{jpeg}, not @file{jpeg.so} or
    453 @file{libextractor_jpeg.so}).
    454 
    455 @example
    456 $ extract -n -l jpeg jpeg_image.jpg
    457 Keywords for file jpeg_image.jpg:
    458 mimetype - image/jpeg
    459 image dimensions - 3x3
    460 comment - (C) 2001 by Christian Grothoff, using gimp 1.2 1
    461 
    462 $ extract -n -l jpeg -x comment jpeg_image.jpg
    463 Keywords for file jpeg_image.jpg:
    464 mimetype - image/jpeg
    465 image dimensions - 3x3
    466 
    467 $ extract -n -l jpeg -p comment jpeg_image.jpg
    468 Keywords for file jpeg_image.jpg:
    469 comment - (C) 2001 by Christian Grothoff, using gimp 1.2 1
    470 
    471 $ extract -n -l gif -p comment jpeg_image.jpg gif_image.gif
    472 Keywords for file jpeg_image.jpg:
    473 Keywords for file gif_image.gif:
    474 comment - Testing keyword extraction
    475 @end example
    476 
    477 
    478 @section Introduction to the libextractor library
    479 
    480 Each public symbol exported by GNU libextractor has the prefix
    481 @verb{|EXTRACTOR_|}.  All-caps names are used for constants.  For the
    482 impatient, the minimal C code for using GNU libextractor (on the
    483 executing binary itself) looks like this:
    484 
    485 @verbatim
    486 #include <extractor.h>
    487 
    488 int 
    489 main (int argc, char ** argv) 
    490 {
    491   struct EXTRACTOR_PluginList *plugins
    492     = EXTRACTOR_plugin_add_defaults (EXTRACTOR_OPTION_DEFAULT_POLICY);
    493   EXTRACTOR_extract (plugins, argv[1],
    494                      NULL, 0, 
    495                      &EXTRACTOR_meta_data_print, stdout);
    496   EXTRACTOR_plugin_remove_all (plugins);
    497   return 0;
    498 }
    499 @end verbatim
    500 
    501 The minimal API illustrated by this example is actually sufficient for
    502 many applications.  The full external C API of GNU libextractor is described
    503 in chapter @xref{Extracting meta data}.  Bindings for other languages
    504 are described in chapter @xref{Language bindings}.  The API for
    505 writing new plugins is described in chapter @xref{Writing new Plugins}.
    506 
    507 Note that it is possible for GNU libextractor to encounter a @code{SIGPIPE}
    508 during its execution.  GNU libextractor --- as it is a library and as such
    509 should not interfere with your main application --- does NOT install a 
    510 signal handler for @code{SIGPIPE}.  You thus need to install a signal
    511 handler (or at least tell your system to ignore @code{SIGPIPE}) if you
    512 want to avoid unexpected problems during calls to GNU libextractor.  
    513 @cindex SIGPIPE
    514 
    515 @node Extracting meta data
    516 @chapter Extracting meta data
    517 
    518 In order to extract meta data with GNU libextractor you first need to
    519 load the respective plugins and then call the extraction API
    520 with the plugins and the data to process.  This section
    521 documents how to load and unload plugins, the various types
    522 and formats in which meta data is returned to the application
    523 and finally the extraction API itself.
    524 
    525 @menu
    526 * Plugin management::   How to load and unload plugins
    527 * Meta types::          About meta types
    528 * Meta formats::        About meta formats
    529 * Extracting::          How to use the extraction API
    530 @end menu
    531 
    532 
    533 @node Plugin management
    534 @section Plugin management
    535 
    536 @cindex reentrant
    537 @cindex concurrency
    538 @cindex threads
    539 @cindex thread-safety
    540 @tindex enum EXTRACTOR_Options
    541 
    542 By default, GNU libextractor runs each plugin in a sub-process, which
    543 isolates the parent process from bugs in the plugins and in the
    544 libraries they use.  On POSIX systems, these sub-processes run the
    545 small program @file{libextractor-helper}, which is installed next to
    546 the plugins and started with @code{posix_spawn()}.  As the helper is
    547 a new program and not a @code{fork()}ed copy of the parent process,
    548 this is safe even if the parent process is multi-threaded.  (Versions
    549 before 1.21 forked the parent.  If another thread of the application
    550 held a lock at that moment, for example in GNU gettext, the child
    551 could deadlock, and on macOS the Objective-C runtime terminated such
    552 children right away.)
    553 
    554 With the @code{EXTRACTOR_OPTION_IN_PROCESS} option, no sub-processes
    555 are used.  In this case, all of the functions for loading and
    556 unloading plugins, including @verb{|EXTRACTOR_plugin_add_defaults|}
    557 and @verb{|EXTRACTOR_plugin_remove_all|}, are thread-safe and
    558 reentrant.
    559 However, using the same plugin list from multiple threads at the same
    560 time is not safe.  
    561 
    562 All plugin code is expected required to be reentrant and state-less,
    563 but due to the extensive use of 3rd party libraries this cannot
    564 be guaranteed.
    565 
    566 
    567 @deftp {C Struct} EXTRACTOR_PluginList
    568 @tindex struct EXTRACTOR_PluginList
    569 
    570 A plugin list represents a set of GNU libextractor plugins.  Most of
    571 the GNU libextractor API is concerned with either constructing a
    572 plugin list or using it to extract meta data.  The internal representation
    573 of the plugin list is of no concern to users or plugin developers.
    574 @end deftp
    575 
    576 
    577 @deftypefun void EXTRACTOR_plugin_remove_all (struct EXTRACTOR_PluginList *plugins)
    578 @findex EXTRACTOR_plugin_remove_all
    579 
    580 Unload all of the plugins in the given list.
    581 @end deftypefun
    582 
    583 @deftypefun {struct EXTRACTOR_PluginList *} EXTRACTOR_plugin_remove (struct EXTRACTOR_PluginList *plugins, const char*name)
    584 @findex EXTRACTOR_plugin_remove
    585 
    586 Unloads a particular plugin.  The given name should be the short name of the plugin, for example ``mime'' for the mime-type extractor or ``mpeg'' for the MPEG extractor.
    587 @end deftypefun
    588 
    589 @deftypefun {struct EXTRACTOR_PluginList *} EXTRACTOR_plugin_add (struct EXTRACTOR_PluginList *plugins, const char* name,const char* options, enum EXTRACTOR_Options flags)
    590 @findex EXTRACTOR_plugin_add
    591 
    592 Loads a particular plugin.  The plugin is added to the existing list, which can be @code{NULL}.  The second argument specifies the name of the plugin (i.e. ``ogg'').  The third argument can be @code{NULL} and specifies plugin-specific options.  Finally, the last argument specifies if the plugin should be executed out-of-process (@code{EXTRACTOR_OPTION_DEFAULT_POLICY}) or not.
    593 @end deftypefun
    594 
    595 @deftypefun {struct EXTRACTOR_PluginList *} EXTRACTOR_plugin_add_config (struct EXTRACTOR_PluginList *plugins, const char* config, enum EXTRACTOR_Options flags)
    596 @findex EXTRACTOR_plugin_add_config
    597 
    598 Loads and unloads plugins based on a configuration string, modifying the existing list, which can be @code{NULL}.  The string has the format ``[-]NAME(OPTIONS)@{:[-]NAME(OPTIONS)@}*''.  Prefixing the plugin name with a ``-'' means that the plugin should be unloaded.
    599 @end deftypefun
    600 
    601 @deftypefun {struct EXTRACTOR_PluginList *} EXTRACTOR_plugin_add_defaults (enum EXTRACTOR_Options flags)
    602 @findex EXTRACTOR_plugin_add_defaults
    603 
    604 Loads all of the plugins in the plugin directory.  This function is what most GNU libextractor applications should use to setup the plugins.
    605 @end deftypefun
    606 
    607 
    608 
    609 @node Meta types
    610 @section Meta types
    611 
    612 
    613 @tindex enum EXTRACTOR_MetaType
    614 @findex EXTRACTOR_metatype_get_max
    615 
    616 @verb{|enum EXTRACTOR_MetaType|} is a C enum which defines a list of over 100 different types of meta data.  The total number can differ between different GNU libextractor releases; the maximum value for the current release can be obtained using the @verb{|EXTRACTOR_metatype_get_max|} function.  All values in this enumeration are of the form @verb{|EXTRACTOR_METATYPE_XXX|}.
    617 
    618 @deftypefun {const char *} EXTRACTOR_metatype_to_string (enum EXTRACTOR_MetaType type)
    619 @findex EXTRACTOR_metatype_to_string
    620 @cindex gettext
    621 @cindex internationalization
    622 
    623 The function @verb{|EXTRACTOR_metatype_to_string|} can be used to obtain a short English string @samp{s} describing the meta data type.  The string can be translated into other languages using GNU gettext with the domain set to GNU libextractor (@verb{|dgettext("libextractor", s)|}).  
    624 @end deftypefun
    625 
    626 @deftypefun {const char *} EXTRACTOR_metatype_to_description (enum EXTRACTOR_MetaType type)
    627 @findex EXTRACTOR_metatype_to_description
    628 @cindex gettext
    629 @cindex internationalization
    630 
    631 The function @verb{|EXTRACTOR_metatype_to_description|} can be used to obtain a longer English string @samp{s} describing the meta data type.  The description may be empty if the short description returned by @code{EXTRACTOR_metatype_to_string} is already comprehensive.  The string can be translated into other languages using GNU gettext with the domain set to GNU libextractor (@verb{|dgettext("libextractor", s)|}).  
    632 @end deftypefun
    633 
    634 
    635 
    636 @node Meta formats
    637 @section Meta formats
    638 
    639 @tindex enum EXTRACTOR_MetaFormat
    640 
    641 @verb{|enum EXTRACTOR_MetaFormat|} is a C enum which defines on a high level how the extracted meta data is represented.  Currently, the library uses three formats: UTF-8 strings, C strings and binary data.  A fourth value, @code{EXTRACTOR_METAFORMAT_UNKNOWN} is defined but not used.  UTF-8 strings are 0-terminated strings that have been converted to UTF-8.  The format code is @code{EXTRACTOR_METAFORMAT_UTF8}. Ideally, most text meta data will be of this format.  Some file formats fail to specify the encoding used for the text.  In this case, the text cannot be converted to UTF-8.  However, the meta data is still known to be 0-terminated and presumably human-readable.  In this case, the format code used is @code{EXTRACTOR_METAFORMAT_C_STRING}; however, this should not be understood to mean that the encoding is the same as that used by the C compiler.  Finally, for binary data (mostly images), the format @code{EXTRACTOR_METAFORMAT_BINARY} is used.
    642 
    643 Naturally this is not a precise description of the meta format. Plugins can provide a more precise description (if known) by providing the respective mime type of the meta data.  For example, binary image meta data could be also tagged as ``image/png'' and normal text would typically be tagged as ``text/plain''.  
    644 
    645 
    646 
    647 @node Extracting
    648 @section Extracting
    649 
    650 @deftypefn {Function Pointer} int (*EXTRACTOR_MetaDataProcessor)(void *cls, const char *plugin_name, enum EXTRACTOR_MetaType type, enum EXTRACTOR_MetaFormat format, const char *data_mime_type, const char *data, size_t data_len)
    651 @tindex EXTRACTOR_MetaDataProcessor
    652 
    653 Type of a function that libextractor calls for each meta data item found.
    654 
    655 @table @var
    656 
    657 @item cls 
    658 closure (user-defined)
    659 
    660 @item plugin_name 
    661 name of the plugin that produced this value; special values can be used (i.e. '<zlib>' for zlib being used in the main libextractor library and yielding meta data);
    662 
    663 @item type 
    664 libextractor-type describing the meta data;
    665 
    666 @item format basic 
    667 format information about data
    668 
    669 @item data_mime_type 
    670 mime-type of data (not of the original file); can be @code{NULL} (if mime-type is not known);
    671 
    672 @item data 
    673 actual meta-data found
    674 
    675 @item data_len 
    676 number of bytes in data
    677 
    678 @end table
    679 
    680 Return 0 to continue extracting, 1 to abort.
    681 @end deftypefn
    682 
    683 
    684 
    685 @deftypefun void EXTRACTOR_extract (struct EXTRACTOR_PluginList *plugins, const char *filename, const void *data, size_t size, EXTRACTOR_MetaDataProcessor proc, void *proc_cls)
    686 @findex EXTRACTOR_extract
    687 @cindex reentrant
    688 @cindex concurrency
    689 @cindex threads
    690 @cindex thread-safety
    691 
    692 This is the main function for extracting keywords with GNU libextractor.  The first argument is a plugin list which specifies the set of plugins that should be used for extracting meta data.  The @samp{filename} argument is optional and can be used to specify the name of a file to process.  If @samp{filename} is @code{NULL}, then the @samp{data} argument must point to the in-memory data to extract meta data from.  If @samp{filename} is non-@code{NULL}, @samp{data} can be @code{NULL}.  If @samp{data} is non-null, then @samp{size} is the size of @samp{data} in bytes.  Otherwise @samp{size} should be zero.  For each meta data item found, GNU libextractor will call the @samp{proc} function, passing @samp{proc_cls} as the first argument to @samp{proc}.  The other arguments to @samp{proc} depend on the specific meta data found.  
    693 
    694 @cindex SIGBUS
    695 @cindex bus error
    696 Meta data extraction should never really fail --- at worst, GNU libextractor should not call @samp{proc} with any meta data. By design, GNU libextractor should never crash or leak memory, even given corrupt files as input.  Note however, that running GNU libextractor on a corrupt file system (or incorrectly @verb{|mmap|}ed files) can result in the operating system sending a SIGBUS (bus error) to the process.  As GNU libextractor typically runs plugins out-of-process, it first maps the file into memory and then attempts to decompress it.  During decompression it is possible to encounter a SIGBUS.   GNU libextractor will @emph{not} attempt to catch this signal and your application is likely to crash.  Note again that this should only happen if the file @emph{system} is corrupt (not if individual files are corrupt).  If this is not acceptable, you might want to consider running GNU libextractor itself also out-of-process (as done, for example, by @url{http://grothoff.org/christian/doodle/,doodle}).
    697 
    698 @end deftypefun
    699 
    700 
    701 @node Language bindings
    702 @chapter Language bindings
    703 @cindex Java
    704 @cindex Mono
    705 @cindex Perl
    706 @cindex Python
    707 @cindex PHP
    708 @cindex Ruby
    709 
    710 GNU libextractor works immediately with C and C++ code. Bindings for Java, Mono, Ruby, Perl, PHP and Python are available for download from the main GNU libextractor website.  Documentation for these bindings (if available) is part of the downloads for the respective binding.  In all cases, a full installation of the C library is required before the binding can be installed.
    711 
    712 @section Java
    713 
    714 Compiling the GNU libextractor Java binding follows the usual process of
    715 running @command{configure} and @command{make}.  The result will be a
    716 shared C library @file{libextractor_java.so} with the native code and
    717 a JAR file (installed to @file{$PREFIX/share/java/libextractor.java}).
    718 
    719 A minimal example for using GNU libextractor's Java binding would look
    720 like this:
    721 @verbatim
    722 import org.gnu.libextractor.*;
    723 import java.util.ArrayList;
    724 
    725 public static void main(String[] args) {
    726   Extractor ex = Extractor.getDefault();
    727   for (int i=0;i<args.length;i++) {
    728     ArrayList keywords = ex.extract(args[i]);
    729     System.out.println("Keywords for " + args[i] + ":");
    730     for (int j=0;j<keywords.size();j++)
    731       System.out.println(keywords.get(j));
    732   }
    733 }
    734 @end verbatim
    735 
    736 The GNU libextractor library and the @file{libextractor_java.so} JNI binding
    737 have to be in the library search path for this to work.  Furthermore, the
    738 @file{libextractor.jar} file should be on the classpath.  
    739 
    740 Note that the API does not use Java 5 style generics in order to work
    741 with older versions of Java.
    742 
    743 @section Mono
    744 
    745 his binding is undocumented at this point.
    746 
    747 @section Perl
    748 
    749 This binding is undocumented at this point.
    750 
    751 @section Python
    752 
    753 This binding is undocumented at this point.
    754 
    755 @section PHP
    756 
    757 This binding is undocumented at this point.
    758 
    759 @section Ruby
    760 
    761 This binding is undocumented at this point.
    762 
    763 
    764 
    765 @node Utility functions
    766 @chapter Utility functions
    767 
    768 @cindex reentrant
    769 @cindex concurrency
    770 @cindex threads
    771 @cindex thread-safety
    772 This chapter describes various utility functions for GNU libextractor usage. All of the functions are reentrant.
    773 
    774 @menu
    775 * Utility Constants::
    776 * Meta data printing::
    777 @end menu
    778 
    779 @node Utility Constants
    780 @section Utility Constants
    781 
    782 @findex EXTRACTOR_VERSION
    783 The constant @verb{|EXTRACTOR_VERSION|} is a hexadecimal
    784 representation of the version number of the installed libextractor
    785 header.  The hexadecimal format is 0xAABBCCDD where AA is the major
    786 version (so far always 0), BB is the minor version, CC is the revision
    787 and DD the patch number.  For example, for version 0.5.18, we would
    788 have AA=0, BB=5, CC=18 and DD=0.  Minor releases such as 0.5.18a or
    789 significant changes in unreleased versions would be marked with DD=1
    790 or higher.
    791 
    792 
    793 @node Meta data printing
    794 @section Meta data printing
    795 
    796 
    797 @findex EXTRACTOR_meta_data_print
    798 The @verb{|EXTRACTOR_meta_data_print|} is a simple function which prints the meta data found with libextractor to a file.  The function is mostly useful for debugging and as an example for how to manipulate the keyword list and can be passed as the @samp{proc} argument to @code{EXTRACTOR_extract}.  The file to print to should be passed as @samp{proc_cls} (which must be of type @code{FILE *}), for example @code{stdout}.
    799 
    800 
    801 
    802 @node Existing Plugins
    803 @chapter Existing Plugins
    804 
    805 @itemize @bullet
    806 @item
    807 APK (Android packages and Java archives, using zlib)
    808 @item
    809 ARCHIVE (using libarchive)
    810 @item
    811 DISKIMAGE (QCOW2, VMDK, VHD and VHDX virtual disk images)
    812 @item
    813 DVI
    814 @item
    815 EBOOK (EPUB and MOBI/AZW)
    816 @item
    817 EXIV2 (using libexiv2, 0.23 or later preferred)
    818 @item 
    819 FLAC (using libFLAC)
    820 @item
    821 GEOTIFF (georeferencing tags in TIFF images)
    822 @item
    823 GPX (GPS tracks)
    824 @item
    825 GIF (using libgif)
    826 @item
    827 GSTREAMER (using libgstreamer v1.0 or later)
    828 @item
    829 HEIF (HEIC, HEIF and AVIF images)
    830 @item
    831 ID3 (MPEG audio with ID3v1/ID3v2 tags, Xing and LAME headers)
    832 @item
    833 ISO9660 (CD/DVD images)
    834 @item
    835 HTML (using libtidy)
    836 @item
    837 IT 
    838 @item
    839 JPEG (using libjpeg v8 or later)
    840 @item
    841 KML (Keyhole Markup Language)
    842 @item
    843 LNK (Windows shell links)
    844 @item
    845 MAN
    846 @item
    847 MIDI (using libsmf)
    848 @item
    849 MIME (using libmagic)
    850 @item
    851 MBOX (Unix mailboxes and RFC 5322 messages)
    852 @item
    853 MPEG (using libmpeg2)
    854 @item
    855 MSOFFICE (OOXML: @file{.docx}, @file{.xlsx}, @file{.pptx}; and
    856 Excel 2-4 BIFF streams)
    857 @item
    858 NSF
    859 @item
    860 PECOFF (Windows executables, DLLs and drivers)
    861 @item
    862 PLIST (Apple property lists, binary and XML)
    863 @item
    864 NSFE
    865 @item
    866 ODF
    867 @item
    868 OLE2 (with libgsf)
    869 @item
    870 OGG (with libogg)
    871 @item
    872 PNG
    873 @item
    874 PS
    875 @item
    876 RIFF
    877 @item
    878 RPM (using librpm)
    879 @item
    880 RTF (@file{.rtf}, including the revision table, comment authors and
    881 custom document properties)
    882 @item
    883 S3M
    884 @item
    885 SQLITE (SQLite 3 databases, write-ahead logs and journals)
    886 @item
    887 SID
    888 @item
    889 PreviewGst (audio preview, using GStreamer)
    890 @item
    891 PreviewOpus (audio preview, using FFmpeg 5.1 or later)
    892 @item
    893 ThumbnailFFMPEG (video thumbnail, using libavformat, libavcodec and
    894 libswscale of FFmpeg 5.1 or later)
    895 @item
    896 ThumbnailGst (video thumbnail, using GStreamer 1.8 or later)
    897 @item
    898 ThumbnailGtk (still image thumbnail, using gdk-pixbuf)
    899 @item
    900 TIFF (with libtiff, tested with v4)
    901 @item
    902 TAR (tar archives)
    903 @item
    904 WAV
    905 @item
    906 WEBP
    907 @item
    908 XM
    909 @item
    910 ZIP
    911 @end itemize
    912 
    913 @file{gzip} and @file{bzip2} compressed versions of these formats are 
    914 also supported (as well as meta data embedded by @file{gzip} itself)
    915 if zlib or libbz2 are available.
    916 
    917 @section Video thumbnails and audio previews
    918 
    919 Four of the plugins in that list do not report meta data that was found
    920 in the file, but meta data computed @emph{from} the file: a small image
    921 taken from a video (@code{EXTRACTOR_METATYPE_THUMBNAIL}) and a short
    922 audio excerpt re-encoded as Opus in an Ogg container
    923 (@code{EXTRACTOR_METATYPE_AUDIO_PREVIEW}).  Both are reported as
    924 @code{EXTRACTOR_METAFORMAT_BINARY}.
    925 
    926 Each feature comes in two implementations, one on top of GStreamer and
    927 one on top of FFmpeg, and which of them are built depends on what
    928 @command{configure} found:
    929 
    930 @table @code
    931 @item thumbnailgst
    932 @itemx thumbnailffmpeg
    933 Decode the video and emit a frame from about 30 seconds in, scaled to
    934 fit a 128x128 box, as JPEG.  Frames that are a perfectly flat surface
    935 (a black leader, a fade-in) are skipped.
    936 @item previewgst
    937 @itemx previewopus
    938 Decode about 15 seconds of audio from about 30 seconds in, fade it in
    939 and out, and emit it as a mono 24 kbit/s Ogg Opus stream with the MIME
    940 type @code{audio/ogg}.
    941 @end table
    942 
    943 In both cases the position is limited to a third of the duration, so a
    944 short file is sampled proportionally earlier.
    945 
    946 If both backends are installed, both plugins run and you get two
    947 thumbnails (or two previews) for the same file.  Which one you want is
    948 an application decision; disable the other one in the configuration
    949 string, as in
    950 
    951 @example
    952 pl = EXTRACTOR_plugin_add_defaults (EXTRACTOR_OPTION_DEFAULT_POLICY);
    953 pl = EXTRACTOR_plugin_add_config (pl,
    954                                   "-thumbnailffmpeg:-previewopus",
    955                                   EXTRACTOR_OPTION_DEFAULT_POLICY);
    956 @end example
    957 
    958 All four accept a comma separated list of options in the plugin
    959 configuration string:
    960 
    961 @table @code
    962 @item size=@var{pixels}
    963 Edge length of the box the thumbnail must fit into (default: 128).
    964 @item format=jpeg|png
    965 Image format of the thumbnail (default: @code{jpeg}).
    966 @item offset=@var{seconds}
    967 How far into the media to sample (default: 30).
    968 @item length=@var{seconds}
    969 Length of the audio preview (default: 15).
    970 @item bitrate=@var{bits-per-second}
    971 Opus bitrate of the audio preview (default: 24000).
    972 @item deadline=@var{milliseconds}
    973 Wall clock budget for the whole extraction; when it runs out, the
    974 plugin reports nothing rather than keeping the application waiting
    975 (default: 10000, or 60000 if @code{srt} or @code{gallery} is given).
    976 @end table
    977 
    978 The two thumbnail plugins also understand:
    979 
    980 @table @code
    981 @item srt:@var{lang}
    982 Additionally dump the complete subtitle track in language @var{lang}
    983 as an SRT file (@code{EXTRACTOR_METATYPE_SUBTITLE_TEXT}, UTF-8, MIME
    984 type @code{application/x-subrip}).  @var{lang} may be given as ISO
    985 639-1 (@code{de}), ISO 639-2/B (@code{ger}) or ISO 639-2/T
    986 (@code{deu}) code; only the first matching track is dumped, and only
    987 text subtitles (SubRip, ASS/SSA, WebVTT, timed text) can be converted.
    988 Without this option the plugins merely list the language of every
    989 subtitle track as @code{EXTRACTOR_METATYPE_SUBTITLE_LANGUAGE}.
    990 @item gallery:@var{n}
    991 Instead of a single thumbnail, emit one image showing @var{n} (at most
    992 100) frames taken at roughly even intervals across the whole video,
    993 arranged in a grid chosen so that the image is as close to 16:9 as
    994 possible.  @code{size} then applies to each tile.  A frame that shows
    995 basically nothing (99% of its pixels within a narrow colour range, such
    996 as a black or white screen) is replaced by one taken a little earlier
    997 or later.
    998 @end table
    999 
   1000 For example, @code{"thumbnailgst(size=256,format=png)"} asks for a
   1001 256x256 PNG, and @code{"thumbnailffmpeg(gallery:12,srt:en)"} for a
   1002 contact sheet of twelve frames plus the English subtitles.
   1003 
   1004 One caveat applies to the GStreamer variants.  Their decoders and
   1005 encoders are themselves GStreamer plugins, resolved at run time, so a
   1006 system that lacks the @code{jpeg} or @code{opus} GStreamer plugin
   1007 builds them happily and then gets no results.
   1008 
   1009 @section Full text
   1010 
   1011 The @code{html}, @code{pdf}, @code{ebook}, @code{odf}, @code{ole2}
   1012 and @code{dvi} plugins can report the entire text of a document as a
   1013 single @code{EXTRACTOR_METATYPE_FULL_TEXT} item in
   1014 @code{EXTRACTOR_METAFORMAT_UTF8} with MIME type @code{text/plain}.  As
   1015 the text can be large, this is only done if the plugin is given the
   1016 option @code{fulltext}, as in
   1017 
   1018 @example
   1019 pl = EXTRACTOR_plugin_add_config (NULL,
   1020                                   "pdf(fulltext):odf(fulltext)",
   1021                                   EXTRACTOR_OPTION_DEFAULT_POLICY);
   1022 @end example
   1023 
   1024 @noindent
   1025 or, on the command line, @code{extract -n -l "pdf(fulltext)" file.pdf}.
   1026 Note the @code{-n}: @code{EXTRACTOR_plugin_add} leaves the list
   1027 unchanged if a plugin of that name is already in it, so adding
   1028 @code{pdf(fulltext)} on top of the default plugin set --- which
   1029 already contains @code{pdf}, without the option --- has no effect at
   1030 all.  Either drop the default set with @code{-n}, or remove the plugin
   1031 before adding it back, as in
   1032 @code{extract -l "-pdf:pdf(fulltext)" file.pdf}.
   1033 
   1034 Within a paragraph, runs of whitespace (including line breaks) are
   1035 reported as a single space; paragraphs are separated by exactly two
   1036 newlines (@code{"\n\n"}).  Where the format makes it possible, text
   1037 that is not part of the running text is left out: page numbers,
   1038 running headers and footers, navigation bars (HTML) and the like.  The
   1039 text is truncated at 16 MiB.
   1040 
   1041 @section Forensic and bulk-scan plugins
   1042 
   1043 Sixteen of the plugins in that list were added for a different job from
   1044 the rest: identifying, and saying something useful about, every file on
   1045 a large volume in a single fast pass.  That purpose shapes how they
   1046 behave, in ways worth knowing about before you use them.
   1047 
   1048 They answer questions about @emph{provenance} rather than about
   1049 content --- which machine, user, tool, device or account a file came
   1050 from.  A Windows executable reports the compiler that built it, the
   1051 path its debug symbols were written to (which routinely discloses the
   1052 build machine's user name) and the exploit mitigations it opts into; a
   1053 shell link reports the volume serial number, the label and the NetBIOS
   1054 name of the machine the target lived on, and often that machine's MAC
   1055 address; a tar archive reports the user and group names of the system
   1056 it was packed on; a mailbox reports the relay chain a message
   1057 travelled and the addresses it passed through.
   1058 
   1059 To keep a bulk scan fast they read headers, not files.  Each rejects a
   1060 file that is not its format within the first few bytes --- which is the
   1061 overwhelmingly common case --- and none of them decodes payload data.
   1062 The @code{sqlite} plugin in particular reads only the 100-byte header
   1063 and never opens the database: opening an untrusted database would be
   1064 both slow and unsafe.  For the same reason none of these plugins links
   1065 a third-party parsing library, so all but @code{apk} and @code{ebook},
   1066 which must inflate members of a zip container, are built
   1067 unconditionally.
   1068 
   1069 Because they characterise rather than enumerate, plugins that meet
   1070 repeating structures --- archive members, mail messages, requested
   1071 permissions, map placemarks --- report a bounded number of them
   1072 together with an @code{EXTRACTOR_METATYPE_ENTRY_COUNT} giving the true
   1073 total.  Do not read the absence of a further item as the absence of
   1074 further items.
   1075 
   1076 Several of the meta types these plugins report were introduced with
   1077 them, and an application that switches on
   1078 @code{enum EXTRACTOR_MetaType} will not know them.  Values are only
   1079 ever appended, so existing values keep their meaning; use
   1080 @code{EXTRACTOR_metatype_get_max()} rather than a hard-coded bound, and
   1081 @code{EXTRACTOR_metatype_to_string()} to render a type you do not
   1082 recognise.
   1083 
   1084 Two habits of these formats are worth remembering when reading their
   1085 output.  Timestamps are reported as ISO 8601 in UTC, but several
   1086 formats store local time with no zone at all --- ISO 9660 and the
   1087 ISO base media format both do --- so a value may be displaced by the
   1088 writer's offset from UTC.  And a field being present does not make it
   1089 true: every value here was written by whatever produced the file, an
   1090 executable's build timestamp is trivially forged, and a disagreement
   1091 between two fields is frequently more informative than either.
   1092 
   1093 @node Writing new Plugins
   1094 @chapter Writing new Plugins
   1095 
   1096 Writing a new plugin for libextractor usually requires writing of or
   1097 interfacing with an actual parser for a specific format.  How this is
   1098 can be accomplished depends on the format and cannot be specified in
   1099 general.  However, care should be taken for the code to be reentrant
   1100 and highly fault-tolerant, especially with respect to malformed
   1101 inputs.
   1102 
   1103 Plugins should start by verifying that the header of the data matches
   1104 the specific format and immediately return if that is not the case.
   1105 Even if the header matches the expected file format, plugins must not
   1106 assume that the remainder of the file is well formed.
   1107 
   1108 The plugin library must be called libextractor_XXX.so, where XXX 
   1109 denotes the file format of the plugin. The library must export a 
   1110 method @verb{|libextractor_XXX_extract_method|}, with the following 
   1111 signature:
   1112 @verbatim
   1113 void
   1114 EXTRACTOR_XXX_extract_method (struct EXTRACTOR_ExtractContext *ec);
   1115 @end verbatim
   1116 
   1117 @samp{ec} contains various information the plugin may need for its
   1118 execution.  Most importantly, it contains functions for reading
   1119 (``read'') and seeking (``seek'') the input data and for returning
   1120 extracted data (``proc'').  The ``config'' member can contain
   1121 additional configuration options.  ``proc'' should be called on
   1122 each meta data item found.  If ``proc'' returns non-zero,
   1123 processing should be aborted (if possible).
   1124 
   1125 In order to test new plugins, the @file{extract} command can be run
   1126 with the options ``-ni'' and ``-l XXX'' .  This will run the plugin
   1127 in-process (making it easier to debug) and without any of the other
   1128 plugins.
   1129 
   1130 
   1131 @section Example for a minimal extract method
   1132 
   1133 The following example shows how a plugin can return the mime type of
   1134 a file.
   1135 @example
   1136 @verbatim
   1137 void
   1138 EXTRACTOR_mymime_extract (struct EXTRACTOR_ExtractContext *ec)
   1139 {
   1140   void *data;
   1141   ssize_t data_size,
   1142 
   1143   if (-1 == (data_size = ec->read (ec->cls, &data, 4)))
   1144     return; /* read error */
   1145   if (data_size < 4)
   1146     return; /* file too small */
   1147   if (0 != memcmp (data, "\177ELF", 4))
   1148     return; /* not ELF */
   1149   if (0 != ec->proc (ec->cls, 
   1150                      "mymime",
   1151                      EXTRACTOR_METATYPE_MIMETYPE,
   1152                      EXTRACTOR_METAFORMAT_UTF8,
   1153                      "text/plain",
   1154                      "application/x-executable",
   1155                      1 + strlen("application/x-executable")))
   1156     return;
   1157   /* more calls to 'proc' here as needed */
   1158 }
   1159 @end verbatim
   1160 @end example
   1161 
   1162 
   1163 @node Internal utility functions
   1164 @chapter Internal utility functions
   1165 
   1166 Some plugins link against the @code{libextractor_common} library which
   1167 provides common abstractions needed by many plugins.  This section
   1168 documents this internal API for plugin developers.  Note that the headers
   1169 for this library are (intentionally) not installed: we do not consider
   1170 this API stable and it should hence only be used by plugins that are 
   1171 build and shipped with GNU libextractor.  Third-party plugins should
   1172 not use it.
   1173 
   1174 @file{convert_numeric.h} defines various conversion functions for
   1175 numbers (in particular, byte-order conversion for floating point
   1176 numbers).  
   1177 
   1178 @file{unzip.h} defines an API for accessing compressed files.
   1179 
   1180 @file{pack.h} provides an interpreter for unpacking structs of integer
   1181 numbers from streams and converting from big or little endian to host
   1182 byte order at the same time.
   1183 
   1184 @file{convert.h} provides a function for character set conversion described
   1185 below.
   1186 
   1187 @deftypefun {char *} EXTRACTOR_common_convert_to_utf8 (const char *input, size_t len, const char *charset)
   1188 @cindex UTF-8
   1189 @cindex character set
   1190 @findex EXTRACTOR_common_convert_to_utf8
   1191 Various GNU libextractor plugins make use of the internal
   1192 @file{convert.h} header which defines a function
   1193 
   1194 @verb{|EXTRACTOR_common_convert_to_utf8|} which can be used to easily convert text from
   1195 any character set to UTF-8.  This conversion is important since the
   1196 linked list of keywords that is returned by GNU libextractor is
   1197 expected to contain only UTF-8 strings.  Naturally, proper conversion
   1198 may not always be possible since some file formats fail to specify the
   1199 character set.  In that case, it is often better to not convert at
   1200 all.
   1201 
   1202 The arguments to @verb{|EXTRACTOR_common_convert_to_utf8|} are the input string (which
   1203 does @emph{not} have to be zero-terminated), the length of the input
   1204 string, and the character set (which @emph{must} be zero-terminated).
   1205 Which character sets are supported depends on the platform, a list can
   1206 generally be obtained using the @command{iconv -l} command.  The
   1207 return value from @verb{|EXTRACTOR_common_convert_to_utf8|} is a zero-terminated string
   1208 in UTF-8 format.  The responsibility to free the string is with the
   1209 caller, so storing the string in the keyword list is acceptable.
   1210 @end deftypefun
   1211 
   1212 
   1213 
   1214 
   1215 
   1216 @node Reporting bugs
   1217 @chapter Reporting bugs
   1218 
   1219 @cindex bug
   1220 GNU libextractor uses the @url{https://gnunet.org/bugs/,Mantis bugtracking
   1221 system}.  If possible, please report bugs there.  You can also e-mail
   1222 the GNU libextractor mailinglist at @url{libextractor@@gnu.org}.
   1223 
   1224 
   1225 
   1226 @c **********************************************************
   1227 @c *******************  Appendices  *************************
   1228 @c **********************************************************
   1229 
   1230 @node GNU Free Documentation License
   1231 @appendix GNU Free Documentation License
   1232 
   1233 @include fdl-1.3.texi
   1234 
   1235 
   1236 @node Index
   1237 @unnumbered Index
   1238 
   1239 @printindex cp
   1240 
   1241 @c @node Function and Data Index
   1242 @c @unnumbered Function and Data Index
   1243 @c @printindex fn
   1244 
   1245 @c @node Type Index
   1246 @c @unnumbered Type Index
   1247 @c @printindex tp
   1248 
   1249 @bye