libextractor.texi (46287B)
1 \input texinfo @c -*- Texinfo -*- 2 @c % The structure of this document is based on the 3 @c % Texinfo manual from libgcrypt by Werner Koch and 4 @c % and Moritz Schulte. 5 @c %**start of header 6 @setfilename libextractor.info 7 @include version.texi 8 @settitle The GNU libextractor Reference Manual 9 @c Unify all the indices into concept index. 10 @syncodeindex fn cp 11 @syncodeindex vr cp 12 @syncodeindex ky cp 13 @syncodeindex pg cp 14 @syncodeindex tp cp 15 @c %**end of header 16 @copying 17 This manual is for GNU libextractor 18 (version @value{VERSION}, @value{UPDATED}), a library for metadata 19 extraction. 20 21 Copyright @copyright{} 2007, 2010, 2012 Christian Grothoff 22 23 @quotation 24 Permission is granted to copy, distribute and/or modify this document 25 under the terms of the GNU Free Documentation License, Version 1.3 26 or any later version published by the Free Software Foundation; 27 with no Invariant Sections, no Front-Cover Texts, and no Back-Cover 28 Texts. A copy of the license is included in the section entitled ``GNU 29 Free Documentation License''. 30 @end quotation 31 @end copying 32 33 @dircategory Software libraries 34 @direntry 35 * Libextractor: (libextractor). Metadata extraction library. 36 @end direntry 37 38 39 40 @c 41 @c Titlepage 42 @c 43 @titlepage 44 @title The GNU libextractor Reference Manual 45 @subtitle Version @value{VERSION} 46 @subtitle @value{UPDATED} 47 @author Christian Grothoff (@email{christian@@grothoff.org}) 48 49 @page 50 @vskip 0pt plus 1filll 51 @insertcopying 52 @end titlepage 53 54 @summarycontents 55 @contents 56 57 58 @ifnottex 59 @node Top 60 @top The GNU libextractor Reference Manual 61 @insertcopying 62 @end ifnottex 63 64 @menu 65 * Introduction:: What is GNU libextractor. 66 * Preparation:: What you should do before using the library. 67 * Generalities:: General library functions and data types. 68 * Extracting meta data:: How to use GNU libextractor to obtain meta data. 69 * Language bindings:: How to use GNU libextractor from languages other than C. 70 * Utility functions:: Utility functions of GNU libextractor. 71 * Existing Plugins:: What plugins are available. 72 * Writing new Plugins:: How to write new plugins for GNU libextractor. 73 * Internal utility functions:: Utility functions of GNU libextractor for writing plugins. 74 * Reporting bugs:: How to report bugs or request new features. 75 76 Appendices 77 78 * GNU Free Documentation License:: Copying this manual. 79 80 Indices 81 82 * Index:: Index 83 @c * Function and Data Index:: Index of functions, variables and data types. 84 @c * Type Index:: Index of data types. 85 86 @end menu 87 88 89 90 @c ********************************************************** 91 @c ******************* Introduction *********************** 92 @c ********************************************************** 93 @node Introduction 94 @chapter Introduction 95 96 @cindex error handling 97 GNU libextractor is GNU's library for extracting meta data from 98 files. Meta data includes format information (such as mime type, 99 image dimensions, color depth, recording frequency), content 100 descriptions (such as document title or document description) and 101 copyright information (such as license, author and contributors). 102 Meta data extraction is an inherently uncertain business --- a parse 103 error can be a corrupt file, an incompatibility in the file format 104 version, an entirely different file format or a bug in the parser. As 105 a result of this uncertainty, GNU libextractor deliberately 106 avoids to ever report any errors. Unexpected file contents simply 107 result in less or possibly no meta data being extracted. 108 109 @cindex plugin 110 GNU libextractor uses plugins to handle various file formats. 111 Technically a plugin can support multiple file formats; however, most 112 plugins only support one particular format. By default, 113 GNU libextractor will use all plugins that are available and found 114 in the plugin installation directory. Applications can 115 request the use of only specific plugins or the exclusion of 116 certain plugins. 117 118 GNU libextractor is distributed with the @command{extract} 119 command@footnote{Some distributions ship @command{extract} in a 120 seperate package.} which is a command-line tool for extracting 121 meta data. @command{extract} is given a list of filenames and 122 prints the resulting meta data to the console. The @command{extract} 123 source code also serves as an advanced example for how to use 124 GNU libextractor. 125 126 This manual focuses on providing documentation for writing software 127 with GNU libextractor. The only relevant parts for end-users 128 are the chapter on compiling and installing GNU libextractor 129 (@xref{Preparation}.). Also, the chapter on existing plugins maybe of 130 interest (@xref{Existing Plugins}.). Additional documentation for 131 end-users can be find in the man page on @command{extract} (using 132 @verb{|man extract|}). 133 134 @cindex license 135 GNU libextractor is licensed under the GNU General Public License, 136 specifically, since version 0.7, GNU libextractor is licensed under GPLv3 137 @emph{or any later version}. 138 139 @node Preparation 140 @chapter Preparation 141 142 This chapter first describes the general build instructions that 143 should apply to all systems. Specific instructions for known problems 144 for particular platforms are then described in individual sections 145 afterwards. 146 147 Compiling GNU libextractor follows the standard GNU autotools build process 148 using @command{configure} and @command{make}. For details on the GNU 149 autotools build process, read the @file{INSTALL} file and query 150 @verb{|./configure --help|} for additional options. 151 152 GNU libextractor has various dependencies, most of which are optional. 153 Instead of specifying the names of the software packages, we 154 will give the list in terms of the names of the respective 155 Debian (wheezy) packages that should be installed. 156 157 You absolutely need: 158 159 @itemize @bullet 160 @item 161 libtool 162 @item 163 gcc 164 @item 165 make 166 @item 167 g++ 168 @item 169 libltdl7-dev 170 @end itemize 171 172 Recommended dependencies are: 173 @itemize @bullet 174 @item 175 zlib1g-dev 176 @item 177 libbz2-dev 178 @item 179 libgif-dev 180 @item 181 libvorbis-dev 182 @item 183 libflac-dev 184 @item 185 libmpeg2-4-dev 186 @item 187 librpm-dev 188 @item 189 libgtk2.0-dev or libgtk3.0-dev 190 @item 191 libgsf-1-dev 192 @item 193 libqt4-dev 194 @item 195 libpoppler-dev 196 @item 197 libexiv2-dev 198 @item 199 libavformat-dev 200 @item 201 libswscale-dev 202 @item 203 libswresample-dev 204 @item 205 libgstreamer1.0-dev 206 @item 207 libgstreamer-plugins-base1.0-dev 208 @end itemize 209 210 For Subversion access and compilation one also needs: 211 @itemize @bullet 212 @item 213 subversion 214 @item 215 autoconf 216 @item 217 automake 218 @end itemize 219 220 Please notify us if we missed some dependencies (note that the list is 221 supposed to only list direct dependencies, not transitive 222 dependencies). 223 224 Once you have compiled and installed GNU libextractor, you should have a file 225 @file{extractor.h} installed in your @file{include/} directory. This 226 file should be the starting point for your C and C++ development with 227 GNU libextractor. The build process also installs the @file{extract} binary and 228 man pages for @file{extract} and GNU libextractor. The @file{extract} man page 229 documents the @file{extract} tool. The GNU libextractor man page gives a brief 230 summary of the C API for GNU libextractor. 231 232 @cindex packageing 233 @cindex directory structure 234 @cindex plugin 235 @cindex environment variables 236 @vindex LIBEXTRACTOR_PREFIX 237 When you install GNU libextractor, various plugins will be 238 installed in the @file{lib/libextractor/} directory. The main library 239 will be installed as @file{lib/libextractor.so}. Note that 240 GNU libextractor will attempt to find the plugins relative to the 241 path of the main library. Consequently, a package manager can move 242 the library and its plugins to a different location later --- as long 243 as the relative path between the main library and the plugins is 244 preserved. As a method of last resort, the user can specify an 245 environment variable @verb{|LIBEXTRACTOR_PREFIX|}. If 246 GNU libextractor cannot locate a plugin, it will look in 247 @verb{|LIBEXTRACTOR_PREFIX/lib/libextractor/|}. 248 249 250 @section Installation on GNU/Linux 251 252 Should work using the standard instructions without problems. 253 254 255 @section Installation on FreeBSD 256 257 Should work using the standard instructions without problems. 258 259 260 @section Installation on OpenBSD 261 262 OpenBSD 3.8 also doesn't have CODESET in @file{langinfo.h}. CODESET 263 is used in GNU libextractor in about three places. This causes problems 264 during compilation. 265 266 267 @section Installation on NetBSD 268 269 No reports so far. 270 271 272 @section Installation using MinGW 273 274 Linking -lstdc++ with the provided libtool fails on Cygwin, this 275 is a problem with libtool, there is unfortunately no flag to tell 276 libtool how to do its job on Cygwin and it seems that it cannot be the 277 default to set the library check to 'pass_all'. Patching libtool may 278 help. 279 280 Note: this is a rather dated report and may no longer apply. 281 282 283 @section Installation on OS X 284 285 libextractor has two installation methods on Mac OS X: it can be 286 installed as a Mac OS X framework or with the standard 287 @command{./configure; make; make install} shell commands. The 288 framework package is self-contained, but currently omits some of the 289 extractor plugins that can be compiled in if libextractor is installed 290 with @command{./configure; make; make install} (provided that the 291 required dependencies exist.) 292 293 @subsection Installing and uninstalling the framework 294 295 The binary framework is distributed as a disk image (@file{Extractor-x.x.xx.dmg}). 296 Installation is done by opening the disk image and clicking @file{Extractor.pkg} 297 inside it. The Mac OS X installer application will then run. The framework 298 is installed to the root volume's @file{/Library/Frameworks} folder and installing 299 will require admin privileges. 300 301 The framework can be uninstalled by dragging @* 302 @file{/Library/Frameworks/Extractor.framework} to the @file{Trash}. 303 304 305 @subsection Using the framework 306 307 In the framework, the @command{extract} command line tool can be found at @* 308 @file{/Library/Frameworks/Extractor.framework/Versions/Current/bin/extract} 309 310 The framework can be used in software projects as a framework or as a dynamic 311 library. 312 313 When using the framework as a dynamic library in projects using autotools, 314 one would most likely want to add @* 315 "-I/Library/Frameworks/Extractor.framework/Versions/Current/include" 316 to CPPFLAGS and @* 317 "-L/Library/Frameworks/Extractor.framework/Versions/Current/lib" 318 to LDFLAGS. 319 320 321 @subsection Example for using the framework 322 323 @example 324 @verbatim 325 // hello.c 326 #include <Extractor/extractor.h> 327 328 int 329 main (int argc, char **argv) 330 { 331 struct EXTRACTOR_PluginList *el; 332 el = EXTRACTOR_plugin_load_defaults (EXTRACTOR_OPTION_DEFAULT_POLICY); 333 // ... 334 EXTRACTOR_plugin_remove_all (el); 335 return 0; 336 } 337 @end verbatim 338 @end example 339 340 You can then compile the example using 341 342 @verbatim 343 $ gcc -o hello hello.c -framework Extractor 344 @end verbatim 345 346 @subsection Example for using the dynamic library 347 348 @example 349 @verbatim 350 // hello.c 351 #include <extractor.h> 352 int main() 353 { 354 struct EXTRACTOR_PluginList *el; 355 el = EXTRACTOR_plugin_load_defaults (EXTRACTOR_OPTION_DEFAULT_POLICY); 356 // ... 357 EXTRACTOR_plugin_remove_all (el); 358 return 0; 359 } 360 @end verbatim 361 @end example 362 363 You can then compile the example using 364 365 @verbatim 366 $ gcc -I/Library/Frameworks/Extractor.framework/Versions/Current/include \ 367 -o hello hello.c \ 368 -L/Library/Frameworks/Extractor.framework/Versions/Current/lib \ 369 -lextractor 370 @end verbatim 371 372 Notice the difference in the @code{#include} line. 373 374 375 376 377 378 379 @section Note to package maintainers 380 381 The suggested way to package GNU libextractor is to split it into 382 roughly the following binary packages: 383 384 @itemize @bullet 385 @item 386 libextractor (main library only, only hard dependency for other packages depending on GNU libextractor) 387 @item 388 extract (command-line tool and man page extract.1) 389 @item 390 libextractor-dev (extractor.h header and man page libextractor.3) 391 @item 392 libextractor-doc (this manual) 393 @item 394 libextractor-plugins (plugins without external dependencies; recommended but not required by extract and libextractor package) 395 @item 396 libextractor-plugin-XXX (plugin with dependency on libXXX, for example for XXX=mpeg this would be @file{libextractor_mpeg.so}) 397 @item 398 libextractor-plugins-all (meta package that requires all plugins except experimental plugins) 399 @end itemize 400 401 This would enable minimal installations (i.e. for embedded systems) to 402 not include any plugins, as well as moderate-size installations (that 403 do not trigger GTK and X11) for systems that have limited resources. 404 Right now, the MP4 plugin is experimental and does nothing and should 405 thus never be included at all; QuickTime, MP4, M4A and 3GPP files are 406 instead handled by the @file{libextractor_qt.so} plugin, which only 407 depends on zlib and is part of libextractor-plugins. The gstreamer plugin is experimental 408 but largely works with the correct version of gstreamer and can thus 409 be packaged (especially if the dependency is available on the target 410 system) but should probably not be part of libextractor-plugins-all. 411 412 413 @node Generalities 414 @chapter Generalities 415 416 @section Introduction to the ``extract'' command 417 418 The @command{extract} command takes a list of file names as arguments, 419 extracts meta data from each of those files and prints the result to 420 the console. By default, @command{extract} will use all available 421 plugins and print all (non-binary) meta data that is found. 422 423 The set of plugins used by @command{extract} can be controlled using 424 the ``-l'' and ``-n'' options. Use ``-n'' to not load all of the 425 default plugins. Use ``-l NAME'' to specifically load a certain 426 plugin. For example, specify ``-n -l mime'' to only use the MIME 427 plugin. 428 429 The argument of ``-l'' is a colon-separated list of plugin names, each 430 optionally preceded by a minus sign to remove the plugin instead of 431 adding it and optionally followed by a plugin configuration in 432 parentheses, as in ``-l "-pdf:pdf(fulltext)"''. Only the last ``-l'' 433 option on the command line is used, so several plugins have to be 434 named in a single list. The options understood by the individual 435 plugins are described in the chapter @emph{Existing Plugins}, under 436 @emph{Video thumbnails and audio previews} and @emph{Full text}. 437 438 Using the ``-p'' option the output of @command{extract} can be limited 439 to only certain keyword types. Similarly, using the ``-x'' option, 440 certain keyword types can be excluded. A list of all known keyword 441 types can be obtained using the ``-L'' option. 442 443 The output format of @command{extract} can be influenced with the 444 ``-V'' (more verbose, lists filenames), ``-g'' (grep-friendly, all 445 meta data on a single line per file) and ``-b'' (bibTeX style) 446 options. 447 448 @section Common usage examples for ``extract'' 449 450 The sample files used here are those of the test suite, in 451 @file{src/plugins/testdata/}. Note that the argument of @code{-l} is 452 the short name of the plugin (@code{jpeg}, not @file{jpeg.so} or 453 @file{libextractor_jpeg.so}). 454 455 @example 456 $ extract -n -l jpeg jpeg_image.jpg 457 Keywords for file jpeg_image.jpg: 458 mimetype - image/jpeg 459 image dimensions - 3x3 460 comment - (C) 2001 by Christian Grothoff, using gimp 1.2 1 461 462 $ extract -n -l jpeg -x comment jpeg_image.jpg 463 Keywords for file jpeg_image.jpg: 464 mimetype - image/jpeg 465 image dimensions - 3x3 466 467 $ extract -n -l jpeg -p comment jpeg_image.jpg 468 Keywords for file jpeg_image.jpg: 469 comment - (C) 2001 by Christian Grothoff, using gimp 1.2 1 470 471 $ extract -n -l gif -p comment jpeg_image.jpg gif_image.gif 472 Keywords for file jpeg_image.jpg: 473 Keywords for file gif_image.gif: 474 comment - Testing keyword extraction 475 @end example 476 477 478 @section Introduction to the libextractor library 479 480 Each public symbol exported by GNU libextractor has the prefix 481 @verb{|EXTRACTOR_|}. All-caps names are used for constants. For the 482 impatient, the minimal C code for using GNU libextractor (on the 483 executing binary itself) looks like this: 484 485 @verbatim 486 #include <extractor.h> 487 488 int 489 main (int argc, char ** argv) 490 { 491 struct EXTRACTOR_PluginList *plugins 492 = EXTRACTOR_plugin_add_defaults (EXTRACTOR_OPTION_DEFAULT_POLICY); 493 EXTRACTOR_extract (plugins, argv[1], 494 NULL, 0, 495 &EXTRACTOR_meta_data_print, stdout); 496 EXTRACTOR_plugin_remove_all (plugins); 497 return 0; 498 } 499 @end verbatim 500 501 The minimal API illustrated by this example is actually sufficient for 502 many applications. The full external C API of GNU libextractor is described 503 in chapter @xref{Extracting meta data}. Bindings for other languages 504 are described in chapter @xref{Language bindings}. The API for 505 writing new plugins is described in chapter @xref{Writing new Plugins}. 506 507 Note that it is possible for GNU libextractor to encounter a @code{SIGPIPE} 508 during its execution. GNU libextractor --- as it is a library and as such 509 should not interfere with your main application --- does NOT install a 510 signal handler for @code{SIGPIPE}. You thus need to install a signal 511 handler (or at least tell your system to ignore @code{SIGPIPE}) if you 512 want to avoid unexpected problems during calls to GNU libextractor. 513 @cindex SIGPIPE 514 515 @node Extracting meta data 516 @chapter Extracting meta data 517 518 In order to extract meta data with GNU libextractor you first need to 519 load the respective plugins and then call the extraction API 520 with the plugins and the data to process. This section 521 documents how to load and unload plugins, the various types 522 and formats in which meta data is returned to the application 523 and finally the extraction API itself. 524 525 @menu 526 * Plugin management:: How to load and unload plugins 527 * Meta types:: About meta types 528 * Meta formats:: About meta formats 529 * Extracting:: How to use the extraction API 530 @end menu 531 532 533 @node Plugin management 534 @section Plugin management 535 536 @cindex reentrant 537 @cindex concurrency 538 @cindex threads 539 @cindex thread-safety 540 @tindex enum EXTRACTOR_Options 541 542 By default, GNU libextractor runs each plugin in a sub-process, which 543 isolates the parent process from bugs in the plugins and in the 544 libraries they use. On POSIX systems, these sub-processes run the 545 small program @file{libextractor-helper}, which is installed next to 546 the plugins and started with @code{posix_spawn()}. As the helper is 547 a new program and not a @code{fork()}ed copy of the parent process, 548 this is safe even if the parent process is multi-threaded. (Versions 549 before 1.21 forked the parent. If another thread of the application 550 held a lock at that moment, for example in GNU gettext, the child 551 could deadlock, and on macOS the Objective-C runtime terminated such 552 children right away.) 553 554 With the @code{EXTRACTOR_OPTION_IN_PROCESS} option, no sub-processes 555 are used. In this case, all of the functions for loading and 556 unloading plugins, including @verb{|EXTRACTOR_plugin_add_defaults|} 557 and @verb{|EXTRACTOR_plugin_remove_all|}, are thread-safe and 558 reentrant. 559 However, using the same plugin list from multiple threads at the same 560 time is not safe. 561 562 All plugin code is expected required to be reentrant and state-less, 563 but due to the extensive use of 3rd party libraries this cannot 564 be guaranteed. 565 566 567 @deftp {C Struct} EXTRACTOR_PluginList 568 @tindex struct EXTRACTOR_PluginList 569 570 A plugin list represents a set of GNU libextractor plugins. Most of 571 the GNU libextractor API is concerned with either constructing a 572 plugin list or using it to extract meta data. The internal representation 573 of the plugin list is of no concern to users or plugin developers. 574 @end deftp 575 576 577 @deftypefun void EXTRACTOR_plugin_remove_all (struct EXTRACTOR_PluginList *plugins) 578 @findex EXTRACTOR_plugin_remove_all 579 580 Unload all of the plugins in the given list. 581 @end deftypefun 582 583 @deftypefun {struct EXTRACTOR_PluginList *} EXTRACTOR_plugin_remove (struct EXTRACTOR_PluginList *plugins, const char*name) 584 @findex EXTRACTOR_plugin_remove 585 586 Unloads a particular plugin. The given name should be the short name of the plugin, for example ``mime'' for the mime-type extractor or ``mpeg'' for the MPEG extractor. 587 @end deftypefun 588 589 @deftypefun {struct EXTRACTOR_PluginList *} EXTRACTOR_plugin_add (struct EXTRACTOR_PluginList *plugins, const char* name,const char* options, enum EXTRACTOR_Options flags) 590 @findex EXTRACTOR_plugin_add 591 592 Loads a particular plugin. The plugin is added to the existing list, which can be @code{NULL}. The second argument specifies the name of the plugin (i.e. ``ogg''). The third argument can be @code{NULL} and specifies plugin-specific options. Finally, the last argument specifies if the plugin should be executed out-of-process (@code{EXTRACTOR_OPTION_DEFAULT_POLICY}) or not. 593 @end deftypefun 594 595 @deftypefun {struct EXTRACTOR_PluginList *} EXTRACTOR_plugin_add_config (struct EXTRACTOR_PluginList *plugins, const char* config, enum EXTRACTOR_Options flags) 596 @findex EXTRACTOR_plugin_add_config 597 598 Loads and unloads plugins based on a configuration string, modifying the existing list, which can be @code{NULL}. The string has the format ``[-]NAME(OPTIONS)@{:[-]NAME(OPTIONS)@}*''. Prefixing the plugin name with a ``-'' means that the plugin should be unloaded. 599 @end deftypefun 600 601 @deftypefun {struct EXTRACTOR_PluginList *} EXTRACTOR_plugin_add_defaults (enum EXTRACTOR_Options flags) 602 @findex EXTRACTOR_plugin_add_defaults 603 604 Loads all of the plugins in the plugin directory. This function is what most GNU libextractor applications should use to setup the plugins. 605 @end deftypefun 606 607 608 609 @node Meta types 610 @section Meta types 611 612 613 @tindex enum EXTRACTOR_MetaType 614 @findex EXTRACTOR_metatype_get_max 615 616 @verb{|enum EXTRACTOR_MetaType|} is a C enum which defines a list of over 100 different types of meta data. The total number can differ between different GNU libextractor releases; the maximum value for the current release can be obtained using the @verb{|EXTRACTOR_metatype_get_max|} function. All values in this enumeration are of the form @verb{|EXTRACTOR_METATYPE_XXX|}. 617 618 @deftypefun {const char *} EXTRACTOR_metatype_to_string (enum EXTRACTOR_MetaType type) 619 @findex EXTRACTOR_metatype_to_string 620 @cindex gettext 621 @cindex internationalization 622 623 The function @verb{|EXTRACTOR_metatype_to_string|} can be used to obtain a short English string @samp{s} describing the meta data type. The string can be translated into other languages using GNU gettext with the domain set to GNU libextractor (@verb{|dgettext("libextractor", s)|}). 624 @end deftypefun 625 626 @deftypefun {const char *} EXTRACTOR_metatype_to_description (enum EXTRACTOR_MetaType type) 627 @findex EXTRACTOR_metatype_to_description 628 @cindex gettext 629 @cindex internationalization 630 631 The function @verb{|EXTRACTOR_metatype_to_description|} can be used to obtain a longer English string @samp{s} describing the meta data type. The description may be empty if the short description returned by @code{EXTRACTOR_metatype_to_string} is already comprehensive. The string can be translated into other languages using GNU gettext with the domain set to GNU libextractor (@verb{|dgettext("libextractor", s)|}). 632 @end deftypefun 633 634 635 636 @node Meta formats 637 @section Meta formats 638 639 @tindex enum EXTRACTOR_MetaFormat 640 641 @verb{|enum EXTRACTOR_MetaFormat|} is a C enum which defines on a high level how the extracted meta data is represented. Currently, the library uses three formats: UTF-8 strings, C strings and binary data. A fourth value, @code{EXTRACTOR_METAFORMAT_UNKNOWN} is defined but not used. UTF-8 strings are 0-terminated strings that have been converted to UTF-8. The format code is @code{EXTRACTOR_METAFORMAT_UTF8}. Ideally, most text meta data will be of this format. Some file formats fail to specify the encoding used for the text. In this case, the text cannot be converted to UTF-8. However, the meta data is still known to be 0-terminated and presumably human-readable. In this case, the format code used is @code{EXTRACTOR_METAFORMAT_C_STRING}; however, this should not be understood to mean that the encoding is the same as that used by the C compiler. Finally, for binary data (mostly images), the format @code{EXTRACTOR_METAFORMAT_BINARY} is used. 642 643 Naturally this is not a precise description of the meta format. Plugins can provide a more precise description (if known) by providing the respective mime type of the meta data. For example, binary image meta data could be also tagged as ``image/png'' and normal text would typically be tagged as ``text/plain''. 644 645 646 647 @node Extracting 648 @section Extracting 649 650 @deftypefn {Function Pointer} int (*EXTRACTOR_MetaDataProcessor)(void *cls, const char *plugin_name, enum EXTRACTOR_MetaType type, enum EXTRACTOR_MetaFormat format, const char *data_mime_type, const char *data, size_t data_len) 651 @tindex EXTRACTOR_MetaDataProcessor 652 653 Type of a function that libextractor calls for each meta data item found. 654 655 @table @var 656 657 @item cls 658 closure (user-defined) 659 660 @item plugin_name 661 name of the plugin that produced this value; special values can be used (i.e. '<zlib>' for zlib being used in the main libextractor library and yielding meta data); 662 663 @item type 664 libextractor-type describing the meta data; 665 666 @item format basic 667 format information about data 668 669 @item data_mime_type 670 mime-type of data (not of the original file); can be @code{NULL} (if mime-type is not known); 671 672 @item data 673 actual meta-data found 674 675 @item data_len 676 number of bytes in data 677 678 @end table 679 680 Return 0 to continue extracting, 1 to abort. 681 @end deftypefn 682 683 684 685 @deftypefun void EXTRACTOR_extract (struct EXTRACTOR_PluginList *plugins, const char *filename, const void *data, size_t size, EXTRACTOR_MetaDataProcessor proc, void *proc_cls) 686 @findex EXTRACTOR_extract 687 @cindex reentrant 688 @cindex concurrency 689 @cindex threads 690 @cindex thread-safety 691 692 This is the main function for extracting keywords with GNU libextractor. The first argument is a plugin list which specifies the set of plugins that should be used for extracting meta data. The @samp{filename} argument is optional and can be used to specify the name of a file to process. If @samp{filename} is @code{NULL}, then the @samp{data} argument must point to the in-memory data to extract meta data from. If @samp{filename} is non-@code{NULL}, @samp{data} can be @code{NULL}. If @samp{data} is non-null, then @samp{size} is the size of @samp{data} in bytes. Otherwise @samp{size} should be zero. For each meta data item found, GNU libextractor will call the @samp{proc} function, passing @samp{proc_cls} as the first argument to @samp{proc}. The other arguments to @samp{proc} depend on the specific meta data found. 693 694 @cindex SIGBUS 695 @cindex bus error 696 Meta data extraction should never really fail --- at worst, GNU libextractor should not call @samp{proc} with any meta data. By design, GNU libextractor should never crash or leak memory, even given corrupt files as input. Note however, that running GNU libextractor on a corrupt file system (or incorrectly @verb{|mmap|}ed files) can result in the operating system sending a SIGBUS (bus error) to the process. As GNU libextractor typically runs plugins out-of-process, it first maps the file into memory and then attempts to decompress it. During decompression it is possible to encounter a SIGBUS. GNU libextractor will @emph{not} attempt to catch this signal and your application is likely to crash. Note again that this should only happen if the file @emph{system} is corrupt (not if individual files are corrupt). If this is not acceptable, you might want to consider running GNU libextractor itself also out-of-process (as done, for example, by @url{http://grothoff.org/christian/doodle/,doodle}). 697 698 @end deftypefun 699 700 701 @node Language bindings 702 @chapter Language bindings 703 @cindex Java 704 @cindex Mono 705 @cindex Perl 706 @cindex Python 707 @cindex PHP 708 @cindex Ruby 709 710 GNU libextractor works immediately with C and C++ code. Bindings for Java, Mono, Ruby, Perl, PHP and Python are available for download from the main GNU libextractor website. Documentation for these bindings (if available) is part of the downloads for the respective binding. In all cases, a full installation of the C library is required before the binding can be installed. 711 712 @section Java 713 714 Compiling the GNU libextractor Java binding follows the usual process of 715 running @command{configure} and @command{make}. The result will be a 716 shared C library @file{libextractor_java.so} with the native code and 717 a JAR file (installed to @file{$PREFIX/share/java/libextractor.java}). 718 719 A minimal example for using GNU libextractor's Java binding would look 720 like this: 721 @verbatim 722 import org.gnu.libextractor.*; 723 import java.util.ArrayList; 724 725 public static void main(String[] args) { 726 Extractor ex = Extractor.getDefault(); 727 for (int i=0;i<args.length;i++) { 728 ArrayList keywords = ex.extract(args[i]); 729 System.out.println("Keywords for " + args[i] + ":"); 730 for (int j=0;j<keywords.size();j++) 731 System.out.println(keywords.get(j)); 732 } 733 } 734 @end verbatim 735 736 The GNU libextractor library and the @file{libextractor_java.so} JNI binding 737 have to be in the library search path for this to work. Furthermore, the 738 @file{libextractor.jar} file should be on the classpath. 739 740 Note that the API does not use Java 5 style generics in order to work 741 with older versions of Java. 742 743 @section Mono 744 745 his binding is undocumented at this point. 746 747 @section Perl 748 749 This binding is undocumented at this point. 750 751 @section Python 752 753 This binding is undocumented at this point. 754 755 @section PHP 756 757 This binding is undocumented at this point. 758 759 @section Ruby 760 761 This binding is undocumented at this point. 762 763 764 765 @node Utility functions 766 @chapter Utility functions 767 768 @cindex reentrant 769 @cindex concurrency 770 @cindex threads 771 @cindex thread-safety 772 This chapter describes various utility functions for GNU libextractor usage. All of the functions are reentrant. 773 774 @menu 775 * Utility Constants:: 776 * Meta data printing:: 777 @end menu 778 779 @node Utility Constants 780 @section Utility Constants 781 782 @findex EXTRACTOR_VERSION 783 The constant @verb{|EXTRACTOR_VERSION|} is a hexadecimal 784 representation of the version number of the installed libextractor 785 header. The hexadecimal format is 0xAABBCCDD where AA is the major 786 version (so far always 0), BB is the minor version, CC is the revision 787 and DD the patch number. For example, for version 0.5.18, we would 788 have AA=0, BB=5, CC=18 and DD=0. Minor releases such as 0.5.18a or 789 significant changes in unreleased versions would be marked with DD=1 790 or higher. 791 792 793 @node Meta data printing 794 @section Meta data printing 795 796 797 @findex EXTRACTOR_meta_data_print 798 The @verb{|EXTRACTOR_meta_data_print|} is a simple function which prints the meta data found with libextractor to a file. The function is mostly useful for debugging and as an example for how to manipulate the keyword list and can be passed as the @samp{proc} argument to @code{EXTRACTOR_extract}. The file to print to should be passed as @samp{proc_cls} (which must be of type @code{FILE *}), for example @code{stdout}. 799 800 801 802 @node Existing Plugins 803 @chapter Existing Plugins 804 805 @itemize @bullet 806 @item 807 APK (Android packages and Java archives, using zlib) 808 @item 809 ARCHIVE (using libarchive) 810 @item 811 DISKIMAGE (QCOW2, VMDK, VHD and VHDX virtual disk images) 812 @item 813 DVI 814 @item 815 EBOOK (EPUB and MOBI/AZW) 816 @item 817 EXIV2 (using libexiv2, 0.23 or later preferred) 818 @item 819 FLAC (using libFLAC) 820 @item 821 GEOTIFF (georeferencing tags in TIFF images) 822 @item 823 GPX (GPS tracks) 824 @item 825 GIF (using libgif) 826 @item 827 GSTREAMER (using libgstreamer v1.0 or later) 828 @item 829 HEIF (HEIC, HEIF and AVIF images) 830 @item 831 ID3 (MPEG audio with ID3v1/ID3v2 tags, Xing and LAME headers) 832 @item 833 ISO9660 (CD/DVD images) 834 @item 835 HTML (using libtidy) 836 @item 837 IT 838 @item 839 JPEG (using libjpeg v8 or later) 840 @item 841 KML (Keyhole Markup Language) 842 @item 843 LNK (Windows shell links) 844 @item 845 MAN 846 @item 847 MIDI (using libsmf) 848 @item 849 MIME (using libmagic) 850 @item 851 MBOX (Unix mailboxes and RFC 5322 messages) 852 @item 853 MPEG (using libmpeg2) 854 @item 855 MSOFFICE (OOXML: @file{.docx}, @file{.xlsx}, @file{.pptx}; and 856 Excel 2-4 BIFF streams) 857 @item 858 NSF 859 @item 860 PECOFF (Windows executables, DLLs and drivers) 861 @item 862 PLIST (Apple property lists, binary and XML) 863 @item 864 NSFE 865 @item 866 ODF 867 @item 868 OLE2 (with libgsf) 869 @item 870 OGG (with libogg) 871 @item 872 PNG 873 @item 874 PS 875 @item 876 RIFF 877 @item 878 RPM (using librpm) 879 @item 880 RTF (@file{.rtf}, including the revision table, comment authors and 881 custom document properties) 882 @item 883 S3M 884 @item 885 SQLITE (SQLite 3 databases, write-ahead logs and journals) 886 @item 887 SID 888 @item 889 PreviewGst (audio preview, using GStreamer) 890 @item 891 PreviewOpus (audio preview, using FFmpeg 5.1 or later) 892 @item 893 ThumbnailFFMPEG (video thumbnail, using libavformat, libavcodec and 894 libswscale of FFmpeg 5.1 or later) 895 @item 896 ThumbnailGst (video thumbnail, using GStreamer 1.8 or later) 897 @item 898 ThumbnailGtk (still image thumbnail, using gdk-pixbuf) 899 @item 900 TIFF (with libtiff, tested with v4) 901 @item 902 TAR (tar archives) 903 @item 904 WAV 905 @item 906 WEBP 907 @item 908 XM 909 @item 910 ZIP 911 @end itemize 912 913 @file{gzip} and @file{bzip2} compressed versions of these formats are 914 also supported (as well as meta data embedded by @file{gzip} itself) 915 if zlib or libbz2 are available. 916 917 @section Video thumbnails and audio previews 918 919 Four of the plugins in that list do not report meta data that was found 920 in the file, but meta data computed @emph{from} the file: a small image 921 taken from a video (@code{EXTRACTOR_METATYPE_THUMBNAIL}) and a short 922 audio excerpt re-encoded as Opus in an Ogg container 923 (@code{EXTRACTOR_METATYPE_AUDIO_PREVIEW}). Both are reported as 924 @code{EXTRACTOR_METAFORMAT_BINARY}. 925 926 Each feature comes in two implementations, one on top of GStreamer and 927 one on top of FFmpeg, and which of them are built depends on what 928 @command{configure} found: 929 930 @table @code 931 @item thumbnailgst 932 @itemx thumbnailffmpeg 933 Decode the video and emit a frame from about 30 seconds in, scaled to 934 fit a 128x128 box, as JPEG. Frames that are a perfectly flat surface 935 (a black leader, a fade-in) are skipped. 936 @item previewgst 937 @itemx previewopus 938 Decode about 15 seconds of audio from about 30 seconds in, fade it in 939 and out, and emit it as a mono 24 kbit/s Ogg Opus stream with the MIME 940 type @code{audio/ogg}. 941 @end table 942 943 In both cases the position is limited to a third of the duration, so a 944 short file is sampled proportionally earlier. 945 946 If both backends are installed, both plugins run and you get two 947 thumbnails (or two previews) for the same file. Which one you want is 948 an application decision; disable the other one in the configuration 949 string, as in 950 951 @example 952 pl = EXTRACTOR_plugin_add_defaults (EXTRACTOR_OPTION_DEFAULT_POLICY); 953 pl = EXTRACTOR_plugin_add_config (pl, 954 "-thumbnailffmpeg:-previewopus", 955 EXTRACTOR_OPTION_DEFAULT_POLICY); 956 @end example 957 958 All four accept a comma separated list of options in the plugin 959 configuration string: 960 961 @table @code 962 @item size=@var{pixels} 963 Edge length of the box the thumbnail must fit into (default: 128). 964 @item format=jpeg|png 965 Image format of the thumbnail (default: @code{jpeg}). 966 @item offset=@var{seconds} 967 How far into the media to sample (default: 30). 968 @item length=@var{seconds} 969 Length of the audio preview (default: 15). 970 @item bitrate=@var{bits-per-second} 971 Opus bitrate of the audio preview (default: 24000). 972 @item deadline=@var{milliseconds} 973 Wall clock budget for the whole extraction; when it runs out, the 974 plugin reports nothing rather than keeping the application waiting 975 (default: 10000, or 60000 if @code{srt} or @code{gallery} is given). 976 @end table 977 978 The two thumbnail plugins also understand: 979 980 @table @code 981 @item srt:@var{lang} 982 Additionally dump the complete subtitle track in language @var{lang} 983 as an SRT file (@code{EXTRACTOR_METATYPE_SUBTITLE_TEXT}, UTF-8, MIME 984 type @code{application/x-subrip}). @var{lang} may be given as ISO 985 639-1 (@code{de}), ISO 639-2/B (@code{ger}) or ISO 639-2/T 986 (@code{deu}) code; only the first matching track is dumped, and only 987 text subtitles (SubRip, ASS/SSA, WebVTT, timed text) can be converted. 988 Without this option the plugins merely list the language of every 989 subtitle track as @code{EXTRACTOR_METATYPE_SUBTITLE_LANGUAGE}. 990 @item gallery:@var{n} 991 Instead of a single thumbnail, emit one image showing @var{n} (at most 992 100) frames taken at roughly even intervals across the whole video, 993 arranged in a grid chosen so that the image is as close to 16:9 as 994 possible. @code{size} then applies to each tile. A frame that shows 995 basically nothing (99% of its pixels within a narrow colour range, such 996 as a black or white screen) is replaced by one taken a little earlier 997 or later. 998 @end table 999 1000 For example, @code{"thumbnailgst(size=256,format=png)"} asks for a 1001 256x256 PNG, and @code{"thumbnailffmpeg(gallery:12,srt:en)"} for a 1002 contact sheet of twelve frames plus the English subtitles. 1003 1004 One caveat applies to the GStreamer variants. Their decoders and 1005 encoders are themselves GStreamer plugins, resolved at run time, so a 1006 system that lacks the @code{jpeg} or @code{opus} GStreamer plugin 1007 builds them happily and then gets no results. 1008 1009 @section Full text 1010 1011 The @code{html}, @code{pdf}, @code{ebook}, @code{odf}, @code{ole2} 1012 and @code{dvi} plugins can report the entire text of a document as a 1013 single @code{EXTRACTOR_METATYPE_FULL_TEXT} item in 1014 @code{EXTRACTOR_METAFORMAT_UTF8} with MIME type @code{text/plain}. As 1015 the text can be large, this is only done if the plugin is given the 1016 option @code{fulltext}, as in 1017 1018 @example 1019 pl = EXTRACTOR_plugin_add_config (NULL, 1020 "pdf(fulltext):odf(fulltext)", 1021 EXTRACTOR_OPTION_DEFAULT_POLICY); 1022 @end example 1023 1024 @noindent 1025 or, on the command line, @code{extract -n -l "pdf(fulltext)" file.pdf}. 1026 Note the @code{-n}: @code{EXTRACTOR_plugin_add} leaves the list 1027 unchanged if a plugin of that name is already in it, so adding 1028 @code{pdf(fulltext)} on top of the default plugin set --- which 1029 already contains @code{pdf}, without the option --- has no effect at 1030 all. Either drop the default set with @code{-n}, or remove the plugin 1031 before adding it back, as in 1032 @code{extract -l "-pdf:pdf(fulltext)" file.pdf}. 1033 1034 Within a paragraph, runs of whitespace (including line breaks) are 1035 reported as a single space; paragraphs are separated by exactly two 1036 newlines (@code{"\n\n"}). Where the format makes it possible, text 1037 that is not part of the running text is left out: page numbers, 1038 running headers and footers, navigation bars (HTML) and the like. The 1039 text is truncated at 16 MiB. 1040 1041 @section Forensic and bulk-scan plugins 1042 1043 Sixteen of the plugins in that list were added for a different job from 1044 the rest: identifying, and saying something useful about, every file on 1045 a large volume in a single fast pass. That purpose shapes how they 1046 behave, in ways worth knowing about before you use them. 1047 1048 They answer questions about @emph{provenance} rather than about 1049 content --- which machine, user, tool, device or account a file came 1050 from. A Windows executable reports the compiler that built it, the 1051 path its debug symbols were written to (which routinely discloses the 1052 build machine's user name) and the exploit mitigations it opts into; a 1053 shell link reports the volume serial number, the label and the NetBIOS 1054 name of the machine the target lived on, and often that machine's MAC 1055 address; a tar archive reports the user and group names of the system 1056 it was packed on; a mailbox reports the relay chain a message 1057 travelled and the addresses it passed through. 1058 1059 To keep a bulk scan fast they read headers, not files. Each rejects a 1060 file that is not its format within the first few bytes --- which is the 1061 overwhelmingly common case --- and none of them decodes payload data. 1062 The @code{sqlite} plugin in particular reads only the 100-byte header 1063 and never opens the database: opening an untrusted database would be 1064 both slow and unsafe. For the same reason none of these plugins links 1065 a third-party parsing library, so all but @code{apk} and @code{ebook}, 1066 which must inflate members of a zip container, are built 1067 unconditionally. 1068 1069 Because they characterise rather than enumerate, plugins that meet 1070 repeating structures --- archive members, mail messages, requested 1071 permissions, map placemarks --- report a bounded number of them 1072 together with an @code{EXTRACTOR_METATYPE_ENTRY_COUNT} giving the true 1073 total. Do not read the absence of a further item as the absence of 1074 further items. 1075 1076 Several of the meta types these plugins report were introduced with 1077 them, and an application that switches on 1078 @code{enum EXTRACTOR_MetaType} will not know them. Values are only 1079 ever appended, so existing values keep their meaning; use 1080 @code{EXTRACTOR_metatype_get_max()} rather than a hard-coded bound, and 1081 @code{EXTRACTOR_metatype_to_string()} to render a type you do not 1082 recognise. 1083 1084 Two habits of these formats are worth remembering when reading their 1085 output. Timestamps are reported as ISO 8601 in UTC, but several 1086 formats store local time with no zone at all --- ISO 9660 and the 1087 ISO base media format both do --- so a value may be displaced by the 1088 writer's offset from UTC. And a field being present does not make it 1089 true: every value here was written by whatever produced the file, an 1090 executable's build timestamp is trivially forged, and a disagreement 1091 between two fields is frequently more informative than either. 1092 1093 @node Writing new Plugins 1094 @chapter Writing new Plugins 1095 1096 Writing a new plugin for libextractor usually requires writing of or 1097 interfacing with an actual parser for a specific format. How this is 1098 can be accomplished depends on the format and cannot be specified in 1099 general. However, care should be taken for the code to be reentrant 1100 and highly fault-tolerant, especially with respect to malformed 1101 inputs. 1102 1103 Plugins should start by verifying that the header of the data matches 1104 the specific format and immediately return if that is not the case. 1105 Even if the header matches the expected file format, plugins must not 1106 assume that the remainder of the file is well formed. 1107 1108 The plugin library must be called libextractor_XXX.so, where XXX 1109 denotes the file format of the plugin. The library must export a 1110 method @verb{|libextractor_XXX_extract_method|}, with the following 1111 signature: 1112 @verbatim 1113 void 1114 EXTRACTOR_XXX_extract_method (struct EXTRACTOR_ExtractContext *ec); 1115 @end verbatim 1116 1117 @samp{ec} contains various information the plugin may need for its 1118 execution. Most importantly, it contains functions for reading 1119 (``read'') and seeking (``seek'') the input data and for returning 1120 extracted data (``proc''). The ``config'' member can contain 1121 additional configuration options. ``proc'' should be called on 1122 each meta data item found. If ``proc'' returns non-zero, 1123 processing should be aborted (if possible). 1124 1125 In order to test new plugins, the @file{extract} command can be run 1126 with the options ``-ni'' and ``-l XXX'' . This will run the plugin 1127 in-process (making it easier to debug) and without any of the other 1128 plugins. 1129 1130 1131 @section Example for a minimal extract method 1132 1133 The following example shows how a plugin can return the mime type of 1134 a file. 1135 @example 1136 @verbatim 1137 void 1138 EXTRACTOR_mymime_extract (struct EXTRACTOR_ExtractContext *ec) 1139 { 1140 void *data; 1141 ssize_t data_size, 1142 1143 if (-1 == (data_size = ec->read (ec->cls, &data, 4))) 1144 return; /* read error */ 1145 if (data_size < 4) 1146 return; /* file too small */ 1147 if (0 != memcmp (data, "\177ELF", 4)) 1148 return; /* not ELF */ 1149 if (0 != ec->proc (ec->cls, 1150 "mymime", 1151 EXTRACTOR_METATYPE_MIMETYPE, 1152 EXTRACTOR_METAFORMAT_UTF8, 1153 "text/plain", 1154 "application/x-executable", 1155 1 + strlen("application/x-executable"))) 1156 return; 1157 /* more calls to 'proc' here as needed */ 1158 } 1159 @end verbatim 1160 @end example 1161 1162 1163 @node Internal utility functions 1164 @chapter Internal utility functions 1165 1166 Some plugins link against the @code{libextractor_common} library which 1167 provides common abstractions needed by many plugins. This section 1168 documents this internal API for plugin developers. Note that the headers 1169 for this library are (intentionally) not installed: we do not consider 1170 this API stable and it should hence only be used by plugins that are 1171 build and shipped with GNU libextractor. Third-party plugins should 1172 not use it. 1173 1174 @file{convert_numeric.h} defines various conversion functions for 1175 numbers (in particular, byte-order conversion for floating point 1176 numbers). 1177 1178 @file{unzip.h} defines an API for accessing compressed files. 1179 1180 @file{pack.h} provides an interpreter for unpacking structs of integer 1181 numbers from streams and converting from big or little endian to host 1182 byte order at the same time. 1183 1184 @file{convert.h} provides a function for character set conversion described 1185 below. 1186 1187 @deftypefun {char *} EXTRACTOR_common_convert_to_utf8 (const char *input, size_t len, const char *charset) 1188 @cindex UTF-8 1189 @cindex character set 1190 @findex EXTRACTOR_common_convert_to_utf8 1191 Various GNU libextractor plugins make use of the internal 1192 @file{convert.h} header which defines a function 1193 1194 @verb{|EXTRACTOR_common_convert_to_utf8|} which can be used to easily convert text from 1195 any character set to UTF-8. This conversion is important since the 1196 linked list of keywords that is returned by GNU libextractor is 1197 expected to contain only UTF-8 strings. Naturally, proper conversion 1198 may not always be possible since some file formats fail to specify the 1199 character set. In that case, it is often better to not convert at 1200 all. 1201 1202 The arguments to @verb{|EXTRACTOR_common_convert_to_utf8|} are the input string (which 1203 does @emph{not} have to be zero-terminated), the length of the input 1204 string, and the character set (which @emph{must} be zero-terminated). 1205 Which character sets are supported depends on the platform, a list can 1206 generally be obtained using the @command{iconv -l} command. The 1207 return value from @verb{|EXTRACTOR_common_convert_to_utf8|} is a zero-terminated string 1208 in UTF-8 format. The responsibility to free the string is with the 1209 caller, so storing the string in the keyword list is acceptable. 1210 @end deftypefun 1211 1212 1213 1214 1215 1216 @node Reporting bugs 1217 @chapter Reporting bugs 1218 1219 @cindex bug 1220 GNU libextractor uses the @url{https://gnunet.org/bugs/,Mantis bugtracking 1221 system}. If possible, please report bugs there. You can also e-mail 1222 the GNU libextractor mailinglist at @url{libextractor@@gnu.org}. 1223 1224 1225 1226 @c ********************************************************** 1227 @c ******************* Appendices ************************* 1228 @c ********************************************************** 1229 1230 @node GNU Free Documentation License 1231 @appendix GNU Free Documentation License 1232 1233 @include fdl-1.3.texi 1234 1235 1236 @node Index 1237 @unnumbered Index 1238 1239 @printindex cp 1240 1241 @c @node Function and Data Index 1242 @c @unnumbered Function and Data Index 1243 @c @printindex fn 1244 1245 @c @node Type Index 1246 @c @unnumbered Type Index 1247 @c @printindex tp 1248 1249 @bye