LIBARCHIVE-FORMATS(5) File Formats Manual LIBARCHIVE-FORMATS(5)
NAME
libarchive-formats — archive formats supported by the libarchive library
DESCRIPTION
The libarchive(3) library reads and writes a variety of streaming archive formats. Generally speaking, all of these archive formats consist of a series of “entries”. Each entry stores a single file system object, such as a file, directory, or symbolic link.
The following provides a brief description of each format supported by libarchive, with some information about recognized extensions or limitations of the current library support. Note that just because a format is supported by libarchive does not imply that a program that uses libarchive will support that format. Applications that use libarchive specify which formats they wish to support, though many programs do use libarchive convenience functions to enable all supported formats.
Tar Formats
The libarchive(3) library can read most tar archives. It can write POSIX-standard “ustar” and “pax interchange” formats as well as v7 tar format and a subset of the legacy GNU tar format.
All tar formats store each entry in one or more 512-byte records. The first record is used for file metadata, including filename, timestamp, and mode information, and the file data is stored in subsequent records. Later variants have extended this by either appropriating undefined areas of the header record, extending the header to multiple records, or by storing special entries that modify the interpretation of subsequent entries.
gnutar
The libarchive(3) library can read most GNU-format tar archives. It currently supports the most popular GNU extensions, including modern long filename and linkname support, as well as atime and ctime data. The libarchive library does not support multi-volume archives, nor the old GNU long filename format. It can read GNU sparse file entries, including the new POSIX-based formats.
The libarchive(3) library can write GNU tar format, including long filename and linkname support, as well as atime and ctime data.
pax
The libarchive(3) library can read and write POSIX-compliant pax interchange format archives. Pax interchange format archives are an extension of the older ustar format that adds a separate entry with additional attributes stored as key/value pairs immediately before each regular entry. The presence of these additional entries is the only difference between pax interchange format and the older ustar format. The extended attributes are of unlimited length and are stored as UTF-8 Unicode strings. Keywords defined in the standard are in all lowercase; vendors are allowed to define custom keys by preceding them with the vendor name in all uppercase. When writing pax archives, libarchive uses many of the SCHILY keys defined by Joerg Schilling’s “star” archiver and a few LIBARCHIVE keys. The libarchive library can read most of the SCHILY keys and most of the GNU keys introduced by GNU tar. It silently ignores any keywords that it does not understand.
The pax interchange format converts filenames to Unicode and stores them using the UTF-8 encoding. Prior to libarchive 3.0, libarchive erroneously assumed that the system wide-character routines natively supported Unicode. This caused it to mis-handle non-ASCII filenames on systems that did not satisfy this assumption.
restricted pax
The libarchive library can also write pax archives in which it attempts to suppress the extended attributes entry whenever possible. The result will be identical to a ustar archive unless the extended attributes entry is required to store a long file name, long linkname, extended ACL, file flags, or if any of the standard ustar data (user name, group name, UID, GID, etc) cannot be fully represented in the ustar header. In all cases, the result can be dearchived by any program that can read POSIX-compliant pax interchange format archives. Programs that correctly read ustar format (see below) will also be able to read this format; any extended attributes will be extracted as separate files stored in PaxHeader directories.
ustar
The libarchive library can both read and write this format. This format has the following limitations:
•
Device major and minor numbers are limited to 21 bits. Nodes with larger numbers will not be added to the archive.
•
Path names in the archive are limited to 255 bytes. (Shorter if there is no / character in exactly the right place.)
•
Symbolic links and hard links are stored in the archive with the name of the referenced file. This name is limited to 100 bytes.
•
Extended attributes, file flags, and other extended security information cannot be stored.
•
Archive entries are limited to 8 gigabytes in size.
Note that the pax interchange format has none of these restrictions. The ustar format is old and widely supported. It is recommended when compatibility is the primary concern.
v7
The libarchive library can read and write the legacy v7 tar format. This format has the following limitations:
•
Only regular files, directories, and symbolic links can be archived. Block and character device nodes, FIFOs, and sockets cannot be archived.
•
Path names in the archive are limited to 100 bytes.
•
Symbolic links and hard links are stored in the archive with the name of the referenced file. This name is limited to 100 bytes.
•
User and group information are stored as numeric IDs; there is no provision for storing user or group names.
•
Extended attributes, file flags, and other extended security information cannot be stored.
•
Archive entries are limited to 8 gigabytes in size.
Generally, users should prefer the ustar format for portability as the v7 tar format is both less useful and less portable.
The libarchive library also reads a variety of commonly-used extensions to the basic tar format. These extensions are recognized automatically whenever they appear.
Numeric extensions.
The POSIX standards require fixed-length numeric fields to be written with some character position reserved for terminators. Libarchive allows these fields to be written without terminator characters. This extends the allowable range; in particular, ustar archives with this extension can support entries up to 64 gigabytes in size. Libarchive also recognizes base-256 values in most numeric fields. This essentially removes all limitations on file size, modification time, and device numbers.
Solaris extensions
Libarchive recognizes ACL and extended attribute records written by Solaris tar.
The first tar program appeared in Seventh Edition Unix in 1979. The first official standard for the tar file format was the “ustar” (Unix Standard Tar) format defined by POSIX in 1988. POSIX.1-2001 extended the ustar format to create the “pax interchange” format.
Cpio Formats
The libarchive library can read and write a number of common cpio variants. A cpio archive stores each entry as a fixed-size header followed by a variable-length filename and variable-length data. Unlike the tar format, the cpio format does only minimal padding of the header or file data. There are several cpio variants, which differ primarily in how they store the initial header: some store the values as octal or hexadecimal numbers in ASCII, others as binary values of varying byte order and length.
binary
The libarchive library transparently reads both big-endian and little-endian variants of the the two binary cpio formats; the original one from PWB/UNIX, and the later, more widely used, variant. This format used 32-bit binary values for file size and mtime, and 16-bit binary values for the other fields. The formats support only the file types present in UNIX at the time of their creation. File sizes are limited to 24 bits in the PWB format, because of the limits of the file system, and to 31 bits in the newer binary format, where signed 32 bit longs were used.
odc
This is the POSIX standardized format, which is officially known as the “cpio interchange format” or the “octet-oriented cpio archive format” and sometimes unofficially referred to as the “old character format”. This format stores the header contents as octal values in ASCII. It is standard, portable, and immune from byte-order confusion. File sizes and mtime are limited to 33 bits (8GB file size), other fields are limited to 18 bits.
SVR4/newc
The libarchive library can read both CRC and non-CRC variants of this format. The SVR4 format uses eight-digit hexadecimal values for all header fields. This limits file size to 4GB, and also limits the mtime and other fields to 32 bits. The SVR4 format can optionally include a CRC of the file contents, although libarchive does not currently verify this CRC.
Cpio first appeared in PWB/UNIX 1.0, which was released within AT&T in 1977. PWB/UNIX 1.0 formed the basis of System III Unix, released outside of AT&T in 1981. This makes cpio older than tar, although cpio was not included in Version 7 AT&T Unix. As a result, the tar command became much better known in universities and research groups that used Version 7. The combination of the find and cpio utilities provided very precise control over file selection. Unfortunately, the format has many limitations that make it unsuitable for widespread use. Only the POSIX format permits files over 4GB, and its 18-bit limit for most other fields makes it unsuitable for modern systems. In addition, cpio formats only store numeric UID/GID values (not usernames and group names), which can make it very difficult to correctly transfer archives across systems with dissimilar user numbering.
Shar Formats
A “shell archive” is a shell script that, when executed on a POSIX-compliant system, will recreate a collection of file system objects. The libarchive library can write two different kinds of shar archives:
shar
The traditional shar format uses a limited set of POSIX commands, including echo(1), mkdir(1), and sed(1). It is suitable for portably archiving small collections of plain text files. However, it is not generally well-suited for large archives (many implementations of sh(1) have limits on the size of a script) nor should it be used with non-text files.
shardump
This format is similar to shar but encodes files using uuencode(1) so that the result will be a plain text file regardless of the file contents. It also includes additional shell commands that attempt to reproduce as many file attributes as possible, including owner, mode, and flags. The additional commands used to restore file attributes make shardump archives less portable than plain shar archives.
ISO9660 format
Libarchive can read and extract from files containing ISO9660-compliant CDROM images. In many cases, this can remove the need to burn a physical CDROM just in order to read the files contained in an ISO9660 image. It also avoids security and complexity issues that come with virtual mounts and loopback devices. Libarchive supports the most common Rockridge extensions and has partial support for Joliet extensions. If both extensions are present, the Joliet extensions will be used and the Rockridge extensions will be ignored. In particular, this can create problems with hardlinks and symlinks, which are supported by Rockridge but not by Joliet.
Libarchive reads ISO9660 images using a streaming strategy. This allows it to read compressed images directly (decompressing on the fly) and allows it to read images directly from network sockets, pipes, and other non-seekable data sources. This strategy works well for optimized ISO9660 images created by many popular programs. Such programs collect all directory information at the beginning of the ISO9660 image so it can be read from a physical disk with a minimum of seeking. However, not all ISO9660 images can be read in this fashion.
Libarchive can also write ISO9660 images. Such images are fully optimized with the directory information preceding all file data. This is done by storing all file data to a temporary file while collecting directory information in memory. When the image is finished, libarchive writes out the directory structure followed by the file data. The location used for the temporary file can be changed by the usual environment variables.
Zip format
Libarchive can read and write zip format archives that have uncompressed entries and entries compressed with the “deflate” , “LZMA” , “XZ” , “BZIP2” and “ZSTD” algorithms. Libarchive can also read, but not write, zip format archives that have entries compressed with the “PPMd” algorithm. Other zip compression algorithms are not supported. The extensions supported by libarchive are Zip64, Libarchive’s extensions to better support streaming, PKZIP’s traditional ZIP encryption, Info-ZIP’s Unix extra fields, extra time, and Unicode path, as well as WinZIP’s AES encryption. It can extract jar archives, __MACOSX resource forks extension for OS X, and self-extracting zip archives. Libarchive can use either of two different strategies for reading Zip archives: a streaming strategy which is fast and can handle extremely large archives, and a seeking strategy which can correctly process self-extracting Zip archives and archives with deleted members or other in-place modifications.
The streaming reader processes Zip archives as they are read. It can read archives of arbitrary size from tape or network sockets, and can decode Zip archives that have been separately compressed or encoded. However, self-extracting Zip archives and archives with certain types of modifications cannot be correctly handled. Such archives require that the reader first process the Central Directory, which is ordinarily located at the end of a Zip archive and is thus inaccessible to the streaming reader. If the program using libarchive has enabled seek support, then libarchive will use this to processes the central directory first.
In particular, the seeking reader must be used to correctly handle self-extracting archives. Such archives consist of a program followed by a regular Zip archive. The streaming reader cannot parse the initial program portion, but the seeking reader starts by reading the Central Directory from the end of the archive. Similarly, Zip archives that have been modified in-place can have deleted entries or other garbage data that can only be accurately detected by first reading the Central Directory.
Archive (library) file format
The Unix archive format (commonly created by the ar(1) archiver) is a general-purpose format which is used almost exclusively for object files to be read by the link editor ld(1). The ar format has never been standardised. There are two common variants: the GNU format derived from SVR4, and the BSD format, which first appeared in 4.4BSD. The two differ primarily in their handling of filenames longer than 15 characters: the GNU/SVR4 variant writes a filename table at the beginning of the archive; the BSD format stores each long filename in an extension area adjacent to the entry. Libarchive can read both extensions, including archives that may include both types of long filenames. Programs using libarchive can write GNU/SVR4 format if they provide an entry called // containing a filename table to be written into the archive before any of the entries. Any entries whose names are not in the filename table will be written using BSD-style long filenames. This can cause problems for programs such as GNU ld that do not support the BSD-style long filenames.
mtree
Libarchive can read and write files in mtree(5) format. This format is not a true archive format, but rather a textual description of a file hierarchy in which each line specifies the name of a file and provides specific metadata about that file. Libarchive can read all of the keywords supported by both the NetBSD and FreeBSD versions of mtree(8), although many of the keywords cannot currently be stored in an archive_entry object. When writing, libarchive supports use of the archive_write_set_options(3) interface to specify which keywords should be included in the output. If libarchive was compiled with access to suitable cryptographic libraries (such as the OpenSSL libraries), it can compute hash entries such as sha512 or md5 from file data being written to the mtree writer.
When reading an mtree file, libarchive will locate the corresponding files on disk using the contents keyword if present or the regular filename. If it can locate and open the file on disk, it will use that to fill in any metadata that is missing from the mtree file and will read the file contents and return those to the program using libarchive. If it cannot locate and open the file on disk, libarchive will return an error for any attempt to read the entry body.
7-Zip
Libarchive can read and write 7-Zip format archives. TODO: Need more information
CAB
Libarchive can read Microsoft Cabinet ( “CAB”) format archives. TODO: Need more information.
LHA
TODO: Information about libarchive’s LHA support
RAR
Libarchive has limited support for reading RAR format archives. Currently, libarchive can read RARv3 format archives which have been either created uncompressed, or compressed using any of the compression methods supported by the RARv3 format. Libarchive can also read self-extracting RAR archives.
Warc
Libarchive can read and write “web archives”. TODO: Need more information
XAR
Libarchive can read and write the XAR format used by many Apple tools. TODO: Need more information
SEE ALSO
ar(1), cpio(1), mkisofs(1), shar(1), tar(1), zip(1), zlib(3), cpio(5), mtree(5), tar(5) Debian December 27, 2016 LIBARCHIVE-FORMATS(5)
LIBARCHIVE-FORMATS(5) BSD File Formats Manual LIBARCHIVE-FORMATS(5)
d28 2 a29 3libarchive-formats — archive formats supported by the libarchive library
d33 6 a38 6The libarchive(3) library reads and writes a variety of streaming archive formats. Generally speaking, all of these archive formats consist of a series of “entries”. Each entry stores a single file system object, such as a file, directory, or symbolic link.
d40 1 a40 1The following d51 2 a52 6
Tar
Formats
The libarchive(3) library can read most tar archives. It can
write POSIX-standard “ustar” and “pax
interchange” formats as well as v7 tar format and a
subset of the legacy GNU tar format.
All tar formats d72 13 a84 13
The libarchive(3) library can read most GNU-format tar archives. It currently supports the most popular GNU extensions, including modern long filename and linkname support, as well as atime and ctime data. The libarchive library does not support multi-volume archives, nor the old GNU long filename format. It can read GNU sparse file entries, including the new POSIX-based formats.
The libarchive(3) library can write GNU tar format, including long filename and linkname support, as well as atime and ctime data.
d88 19 a106 18The libarchive(3) library can read and write POSIX-compliant pax interchange format archives. Pax interchange format archives are an extension of the older ustar format that adds a separate entry with additional attributes stored as key/value pairs immediately before each regular entry. The presence of these additional entries is the only difference between pax interchange format and the older ustar format. The extended attributes are of unlimited length and are stored as UTF-8 Unicode strings. Keywords defined in the standard are in all lowercase; vendors are allowed to define custom keys by preceding them with the vendor name in all uppercase. When writing pax archives, libarchive uses many of the SCHILY keys defined by Joerg Schilling’s “star” archiver and a few LIBARCHIVE keys. The libarchive library can read most of the SCHILY keys and most of the GNU keys introduced by GNU tar. It silently ignores any keywords that it does not understand.
d108 1 a108 1The pax d118 1 a118 1
The libarchive library can also d135 1 a135 1
The libarchive d141 1 a141 1
Device major and minor numbers d147 1 a147 1
Path names in the archive are d153 1 a153 1
Symbolic links and hard links d159 1 a159 1
Extended attributes, file d165 1 a165 1
Archive entries are limited to d168 1 a168 1
Note that the pax interchange d175 1 a175 1
The libarchive d181 1 a181 1
Only regular files, d188 1 a188 1
Path names in the archive are d193 1 a193 1
Symbolic links and hard links d199 1 a199 1
User and group information are d205 1 a205 1
Extended attributes, file d211 1 a211 1
Archive entries are limited to d214 1 a214 1
Generally, users should prefer d218 1 a218 1
The libarchive d225 1 a225 1
The POSIX standards require d238 1 a238 1
Libarchive recognizes ACL and d241 1 a241 1
The first tar d248 13 a260 11
Cpio
Formats
The libarchive library can read and write a number of common
cpio variants. A cpio archive stores each entry as a
fixed-size header followed by a variable-length filename and
variable-length data. Unlike the tar format, the cpio format
does only minimal padding of the header or file data. There
are several cpio variants, which differ primarily in how
they store the initial header: some store the values as
octal or hexadecimal numbers in ASCII, others as binary
values of varying byte order and length.
The libarchive d278 1 a278 1
This is the d291 1 a291 1
The libarchive library can read d299 1 a299 1
Cpio first d317 8 a324 6
Shar
Formats
A “shell archive” is a shell script that, when
executed on a POSIX-compliant system, will recreate a
collection of file system objects. The libarchive library
can write two different kinds of shar archives:
The traditional d330 6 a335 5 echo(1), mkdir(1), and sed(1). It is suitable for portably archiving small collections of plain text files. However, it is not generally well-suited for large archives (many implementations of sh(1) have limits on the size of a script) nor should it be used with non-text files.
d339 24 a362 23This format is similar to shar but encodes files using uuencode(1) so that the result will be a plain text file regardless of the file contents. It also includes additional shell commands that attempt to reproduce as many file attributes as possible, including owner, mode, and flags. The additional commands used to restore file attributes make shardump archives less portable than plain shar archives.
ISO9660
format
Libarchive can read and extract from files containing
ISO9660-compliant CDROM images. In many cases, this can
remove the need to burn a physical CDROM just in order to
read the files contained in an ISO9660 image. It also avoids
security and complexity issues that come with virtual mounts
and loopback devices. Libarchive supports the most common
Rockridge extensions and has partial support for Joliet
extensions. If both extensions are present, the Joliet
extensions will be used and the Rockridge extensions will be
ignored. In particular, this can create problems with
hardlinks and symlinks, which are supported by Rockridge but
not by Joliet.
Libarchive reads d376 1 a376 1
Libarchive can d386 2 a387 13
Zip
format
Libarchive can read and write zip format archives that have
uncompressed entries and entries compressed with the
“deflate” algorithm. Other zip compression
algorithms are not supported. It can extract jar archives,
archives that use Zip64 extensions and self-extracting zip
archives. Libarchive can use either of two different
strategies for reading Zip archives: a streaming strategy
which is fast and can handle extremely large archives, and a
seeking strategy which can correctly process self-extracting
Zip archives and archives with deleted members or other
in-place modifications.
The streaming d414 1 a414 1
In particular, d425 13 a437 11
Archive
(library) file format
The Unix archive format (commonly created by the ar(1)
archiver) is a general-purpose format which is used almost
exclusively for object files to be read by the link editor
ld(1). The ar format has never been standardised. There are
two common variants: the GNU format derived from SVR4, and
the BSD format, which first appeared in 4.4BSD. The two
differ primarily in their handling of filenames longer than
15 characters: the GNU/SVR4 variant writes a filename table
at the beginning of the archive; the BSD format stores each
d449 13
a461 11
mtree
Libarchive can read and write files in mtree(5) format. This
format is not a true archive format, but rather a textual
description of a file hierarchy in which each line specifies
the name of a file and provides specific metadata about that
file. Libarchive can read all of the keywords supported by
both the NetBSD and FreeBSD versions of mtree(8), although
many of the keywords cannot currently be stored in an
archive_entry object. When writing, libarchive supports use
of the archive_write_set_options(3) interface to specify
d468 1
a468 1
When reading an d478 36 a513 27
7-Zip
Libarchive can read and write 7-Zip format archives. TODO:
Need more information
CAB
Libarchive can read Microsoft Cabinet ( “CAB”)
format archives. TODO: Need more information.
LHA
TODO: Information about libarchive’s LHA support
RAR
Libarchive has limited support for reading RAR format
archives. Currently, libarchive can read RARv3 format
archives which have been either created uncompressed, or
compressed using any of the compression methods supported by
the RARv3 format. Libarchive can also read self-extracting
RAR archives.
Warc
Libarchive can read and write “web archives”.
TODO: Need more information
XAR
Libarchive can read and write the XAR format used by many
Apple tools. TODO: Need more information
ar(1), cpio(1), mkisofs(1), shar(1), tar(1), zip(1), zlib(3), cpio(5), mtree(5), tar(5)
BSD December 27, 2016 BSD
@ 1.11 log @libarchive: updated to 3.7.4 Libarchive 3.7.4 is a bugfix and security release Security fixes: rar: Fix OOB in rar e8 filter (CVE-2024-26256) zip: Fix out of boundary access Important bugfixes: 7zip: Limit amount of properties bsdtar: Fix error handling around strtol() usages passphrase: Improve newline handling on Windows passphrase: Never allow empty passwords rar: Fix "File CRC Error" when extracting specific rar4 archives xar: Avoid infinite link loop zip: Update AppleDouble support for directories zstd: Implement core detection @ text @d2 1 a2 1 @ 1.10 log @libarchive: updated to 3.7.3 Libarchive 3.7.3 is a feature, security and bugfix release. New features: PCRE2 support add trailing letter b to bsdtar(1) substitute pattern add support for long options "--group" and "--owner" to tar(1) Security fixes: Fix possible vulnerability in tar error reporting introduced in f27c173 Important bugfixes: ISO9660: preserve the natural order of links rar5: fix decoding unicode filenames on Windows rar5: fix infinite loop if during rar5 decompression the last block produced no data xz filter: fix incorrect eof at the end of an lzip member zip: fix end-of-data marker processing when decompressing zip archives multiple bsdunzip(1) fixes filetime truncation fix on Windows @ text @d2 1 a2 1 @ 1.9 log @libarchive: updated to 3.7.2 Libarchive 3.7.2 is a security, bugfix and feature release. Security fixes: Multiple vulnerabilities have been fixed in the PAX writer (1b4e0d0) Important bugfixes: bsdunzip(1) now correctly handles arguments following an -x after the zipfile New features: bsdunzip(1) now supports the "--version" flag 7-zip reader now translates Windows permissions into UNIX permissions uudecode filter in raw mode now supports file name and file mode zstd filter now supports the "long" write option Libarchive 3.7.1 is a security, feature and bugfix release. Security fixes: SEGV and stack buffer overflow in verbose mode of cpio Feature updates: bsdunzip updated to match latest upstream code Important bugfixes: miscellaneous functional bugfixes build fixes on multiple platforms Libarchive 3.7.0 is a feature and bugfix release. New features: bsdunzip: new tool ported from FreeBSD drop-in replacement for Info-ZIP unzip, not yet ported for Windows 7zip reader: support for Zstandard compression 7zip reader: support for ARM64 filter zstd filter: support for multi-frame zstd archives Other notable bugfixes and improvements: pax: fix year 2038 problem on platforms with 64-bit time_t Windows: Universal Windows Platform (UWP) fixes and improvements Windows: bcrypt usage fixes and improvements Windows: time function usage fixes and improvements @ text @d2 1 a2 1 @ 1.8 log @libarchive: Update to 3.4.3 Libarchive 3.4.3 is a feature and bugfix release. New features: support for pzstd compressed files (#1357) support for RHT.security.selinux tar extended attribute (#1348) Important bugfixes: various zstd fixes and improvements (#1342 #1352 #1359) child process handling fixes (#1372) Libarchive 3.4.2 is a feature and security release. New features: support for atomic file extraction (bsdtar -x --safe-writes) (#1289) support for mbed TLS (PolarSSL) (#1301) Important bugfixes: security fixes in RAR5 reader (#1280 #1326) compression buffer fix in XAR writer (#1317) fix uname and gname longer than 32 characters in PAX writer (#1319) fix segfault when archiving hard links in ISO9660 and XAR writers (#1325) fix support for extracting 7z archive entries with Delta filter (#987) Libarchive 3.4.1 is a feature and security release. New features: Unicode filename support for reading lha/lzh archives New pax write option "xattrhdr" Important bugfixes: security fixes in wide string processing (#1276 #1298) security fixes in RAR5 reader (#1212 #1217 #1296) security fixes and optimizations to write filter logic (#351) security fix related to use of readlink(2) (1dae5a5) sparse file handling fixes (#1218 #1260) Thanks to all contributors and bug reporters. Special thanks to Christos Zoulas (@@zoulasc) from NetBSD for the atomic file extraction feature. @ text @d1 2 a2 2 d37 3 a39 3 of ’’entries’’. Each entry stores a single file system object, such as a file, directory, or symbolic link. d55 3 a57 4 write POSIX-standard ’’ustar’’ and ’’pax interchange’’ formats as well as v7 tar format and a subset of the legacy GNU tar format. d101 4 a104 5 ’’star’’ archiver and a few LIBARCHIVE keys. The libarchive library can read most of the SCHILY keys and most of the GNU keys introduced by GNU tar. It silently ignores any keywords that it does not understand. d242 3 a244 4 ’’ustar’’ (Unix Standard Tar) format defined by POSIX in 1988. POSIX.1-2001 extended the ustar format to create the ’’pax interchange’’ format. d248 9 a256 11 The libarchive library can read a number of common cpio variants and can write ’’odc’’ and ’’newc’’ format archives. A cpio archive stores each entry as a fixed-size header followed by a variable-length filename and variable-length data. Unlike the tar format, the cpio format does only minimal padding of the header or file data. There are several cpio variants, which differ primarily in how they store the initial header: some store the values as octal or hexadecimal numbers in ASCII, others as binary values of varying byte order and length. d262 9 a270 3 little-endian variants of the original binary cpio format. This format used 32-bit binary values for file size and mtime, and 16-bit binary values for the other fields. d274 10 a283 11The libarchive library can both read and write this POSIX-standard format, which is officially known as the ’’cpio interchange format’’ or the ’’octet-oriented cpio archive format’’ and sometimes unofficially referred to as the ’’old character format’’. This format stores the header contents as octal values in ASCII. It is standard, portable, and immune from byte-order confusion. File sizes and mtime are limited to 33 bits (8GB file size), other fields are limited to 18 bits.
d315 4 a318 4 A ’’shell archive’’ is a shell script that, when executed on a POSIX-compliant system, will recreate a collection of file system objects. The libarchive library can write two different kinds of shar archives: d382 9 a390 9 ’’deflate’’ algorithm. Other zip compression algorithms are not supported. It can extract jar archives, archives that use Zip64 extensions and self-extracting zip archives. Libarchive can use either of two different strategies for reading Zip archives: a streaming strategy which is fast and can handle extremely large archives, and a seeking strategy which can correctly process self-extracting Zip archives and archives with deleted members or other in-place modifications. d471 2 a472 3 Libarchive can read Microsoft Cabinet ( ’’CAB’’) format archives. TODO: Need more information. d486 2 a487 2 Libarchive can read and write ’’web archives’’. TODO: Need more information @ 1.7 log @Update for libarchive-3.4.0: - improvements for Android APK and JAR archives - better support for non-recursive list and extract - tar --exclude-vcs support - fixes for file attributes and flags handling - zipx support - rar 5.0 reader @ text @d1 2 a2 2 d37 3 a39 3 of “entries”. Each entry stores a single file system object, such as a file, directory, or symbolic link. d55 4 a58 3 write POSIX-standard “ustar” and “pax interchange” formats as well as v7 tar format and a subset of the legacy GNU tar format. d102 5 a106 4 “star” archiver and a few LIBARCHIVE keys. The libarchive library can read most of the SCHILY keys and most of the GNU keys introduced by GNU tar. It silently ignores any keywords that it does not understand. d244 4 a247 3 “ustar” (Unix Standard Tar) format defined by POSIX in 1988. POSIX.1-2001 extended the ustar format to create the “pax interchange” format. d252 4 a255 4 variants and can write “odc” and “newc” format archives. A cpio archive stores each entry as a fixed-size header followed by a variable-length filename and variable-length data. Unlike d275 9 a283 8 which is officially known as the “cpio interchange format” or the “octet-oriented cpio archive format” and sometimes unofficially referred to as the “old character format”. This format stores the header contents as octal values in ASCII. It is standard, portable, and immune from byte-order confusion. File sizes and mtime are limited to 33 bits (8GB file size), other fields are limited to 18 bits. d315 4 a318 4 A “shell archive” is a shell script that, when executed on a POSIX-compliant system, will recreate a collection of file system objects. The libarchive library can write two different kinds of shar archives: d382 9 a390 9 “deflate” algorithm. Other zip compression algorithms are not supported. It can extract jar archives, archives that use Zip64 extensions and self-extracting zip archives. Libarchive can use either of two different strategies for reading Zip archives: a streaming strategy which is fast and can handle extremely large archives, and a seeking strategy which can correctly process self-extracting Zip archives and archives with deleted members or other in-place modifications. d471 3 a473 2 Libarchive can read Microsoft Cabinet ( “CAB”) format archives. TODO: Need more information. d487 2 a488 2 Libarchive can read and write “web archives”. TODO: Need more information @ 1.6 log @libarchive: updated to 3.3.3 libarchive 3.3.3: Avoid super-linear slowdown on malformed mtree files Many fixes for building with Visual Studio NO_OVERWRITE doesn't change existing directory attributes New support for Zstandard read and write filters @ text @d1 2 a2 2 d37 3 a39 3 of ’’entries’’. Each entry stores a single file system object, such as a file, directory, or symbolic link. d55 3 a57 4 write POSIX-standard ’’ustar’’ and ’’pax interchange’’ formats as well as v7 tar format and a subset of the legacy GNU tar format. d101 4 a104 5 ’’star’’ archiver and a few LIBARCHIVE keys. The libarchive library can read most of the SCHILY keys and most of the GNU keys introduced by GNU tar. It silently ignores any keywords that it does not understand. d242 3 a244 4 ’’ustar’’ (Unix Standard Tar) format defined by POSIX in 1988. POSIX.1-2001 extended the ustar format to create the ’’pax interchange’’ format. d249 4 a252 4 variants and can write ’’odc’’ and ’’newc’’ format archives. A cpio archive stores each entry as a fixed-size header followed by a variable-length filename and variable-length data. Unlike d272 8 a279 9 which is officially known as the ’’cpio interchange format’’ or the ’’octet-oriented cpio archive format’’ and sometimes unofficially referred to as the ’’old character format’’. This format stores the header contents as octal values in ASCII. It is standard, portable, and immune from byte-order confusion. File sizes and mtime are limited to 33 bits (8GB file size), other fields are limited to 18 bits. d311 4 a314 4 A ’’shell archive’’ is a shell script that, when executed on a POSIX-compliant system, will recreate a collection of file system objects. The libarchive library can write two different kinds of shar archives: d378 9 a386 9 ’’deflate’’ algorithm. Other zip compression algorithms are not supported. It can extract jar archives, archives that use Zip64 extensions and self-extracting zip archives. Libarchive can use either of two different strategies for reading Zip archives: a streaming strategy which is fast and can handle extremely large archives, and a seeking strategy which can correctly process self-extracting Zip archives and archives with deleted members or other in-place modifications. d467 2 a468 3 Libarchive can read Microsoft Cabinet ( ’’CAB’’) format archives. TODO: Need more information. d482 2 a483 2 Libarchive can read and write ’’web archives’’. TODO: Need more information @ 1.5 log @Merge for libarchive-3.3.2. @ text @d2 1 a2 1 @ 1.4 log @Merge libarchive-3.3.1. @ text @d2 1 a2 1 d37 1 a37 1 of ‘‘entries’’. Each entry stores a d55 2 a56 2 write POSIX-standard ‘‘ustar’’ and ‘‘pax interchange’’ formats as well d102 1 a102 1 ‘‘star’’ archiver and a few d244 1 a244 1 ‘‘ustar’’ (Unix Standard Tar) format d246 1 a246 1 format to create the ‘‘pax d252 2 a253 2 variants and can write ‘‘odc’’ and ‘‘newc’’ format archives. A cpio d275 1 a275 1 which is officially known as the ‘‘cpio d277 1 a277 1 ‘‘octet-oriented cpio archive d279 1 a279 1 as the ‘‘old character format’’. d315 1 a315 1 A ‘‘shell archive’’ is a shell d382 1 a382 1 ‘‘deflate’’ algorithm. Other zip d472 1 a472 1 ‘‘CAB’’) format archives. TODO: Need d487 1 a487 1 Libarchive can read and write ‘‘web @ 1.3 log @Update for libarchive 3.2.1. @ text @d2 1 a2 1 d37 1 a37 1 of ’’entries’’. Each entry stores a d55 2 a56 2 write POSIX-standard ’’ustar’’ and ’’pax interchange’’ formats as well d102 1 a102 1 ’’star’’ archiver and a few d239 1 a239 3 extended attribute records written by Solaris tar. Currently, libarchive only has support for old-style ACLs; the newer NFSv4 ACLs are recognized but discarded. d244 1 a244 1 ’’ustar’’ (Unix Standard Tar) format d246 1 a246 1 format to create the ’’pax d252 2 a253 2 variants and can write ’’odc’’ and ’’newc’’ format archives. A cpio d275 1 a275 1 which is officially known as the ’’cpio d277 1 a277 1 ’’octet-oriented cpio archive d279 1 a279 1 as the ’’old character format’’. d315 1 a315 1 A ’’shell archive’’ is a shell d382 1 a382 1 ’’deflate’’ algorithm. Other zip d472 1 a472 1 ’’CAB’’) format archives. TODO: Need d487 1 a487 1 Libarchive can read and write ’’web d501 1 a501 1 March 18, 2012 BSD @ 1.2 log @Changes 3.1.2: This is a maintenance update to fix issues with the new RAR seeking feature. This new release also contains fixes for build failures when building libarchive using Visual Studio 2012 and MinGW. @ text @d1 2 a2 2 d56 3 a58 2 ’’pax interchange’’ formats and a subset of the legacy GNU tar format. d173 45 d287 1 a287 1SVR4
d289 7 a295 7The libarchive library can read both CRC and non-CRC variants of this format. The SVR4 format uses eight-digit hexadecimal values for all header fields. This limits file size to 4GB, and also limits the mtime and other fields to 32 bits. The SVR4 format can optionally include a CRC of the file contents, although libarchive does not currently verify this CRC.
d432 7 a438 6 libarchive can write GNU/SVR4 format if they provide a filename table to be written into the archive before any of the entries. Any entries whose names are not in the filename table will be written using BSD-style long filenames. This can cause problems for programs such as GNU ld that do not support the BSD-style long filenames. d467 4 a470 2LHA
XXX Information about libarchive’s LHA support XXX
XAR
XXX Information about libarchive’s XAR support XXX
libarchive-formats(5) FreeBSD File Formats Manual libarchive-formats(5)
d26 1 a26 1NAME
d28 1 a28 1libarchive-formats d32 1 d34 1 a34 3
DESCRIPTION
The libarchive(3) library reads d37 1 a37 1 of ‘‘entries’’. Each entry stores a d41 1 a41 1
The following d52 1 a52 1
Tar d54 4 a57 4 The libarchive(3) library can read most tar archives. However, it only writes POSIX-standard ‘‘ustar’’ and ‘‘pax interchange’’ formats.
d59 1 a59 1All tar formats d69 1 a69 1
gnutar
d71 3 a73 3The libarchive(3) library can read GNU-format tar archives. It currently supports the most popular GNU extensions, d78 6 a83 2 new POSIX-based formats, but cannot write GNU sparse file entries.
d85 1 a85 1pax
d87 1 a87 1The d101 1 a101 1 ‘‘star’’ archiver and a few d107 7 a113 2
restricted pax
d115 3 a117 1The libarchive library can also d132 1 a132 1
ustar
d134 1 a134 1The libarchive d138 1 a138 1
•
d140 1 a140 1Device major and minor numbers d144 1 a144 1
•
d146 1 a146 1Path names in the archive are d150 1 a150 1
•
d152 1 a152 1Symbolic links and hard links d156 1 a156 1
•
d158 1 a158 1Extended attributes, file d162 1 a162 1
•
d164 1 a164 1Archive entries are limited to d167 4 a170 2
Note that the pax interchange format has none of these restrictions.
d172 1 a172 1The libarchive d177 1 a177 2
Numeric extensions.
d179 1 a179 1The POSIX standards require d190 1 a190 2
Solaris extensions
d192 1 a192 1Libarchive recognizes ACL and d197 1 a197 1
The first tar d200 1 a200 1 ‘‘ustar’’ (Unix Standard Tar) format d202 1 a202 1 format to create the ‘‘pax d205 1 a205 1
Cpio d208 2 a209 2 variants and can write ‘‘odc’’ and ‘‘newc’’ format archives. A cpio d219 1 a219 1
binary
d221 1 a221 1The libarchive d227 1 a227 1
odc
d229 1 a229 1The libarchive d231 1 a231 1 which is officially known as the ‘‘cpio d233 1 a233 1 ‘‘octet-oriented cpio archive d235 1 a235 1 as the ‘‘old character format’’. d241 1 a241 1
SVR4
d243 1 a243 1The libarchive d251 1 a251 1
Cpio first d269 1 a269 1
Shar d271 1 a271 1 A ‘‘shell archive’’ is a shell d276 1 a276 1
shar
d278 1 a278 1The traditional d286 1 d288 1 a288 3
shardump
This format is similar to shar d297 1 a297 1
ISO9660 d312 23 a334 1
Zip d338 1 a338 1 ‘‘deflate’’ algorithm. Older zip d340 31 a370 7 archives, archives that use Zip64 extensions and many self-extracting zip archives. Libarchive reads Zip archives as they are being streamed, which allows it to read archives of arbitrary size. It currently does not use the central directory; this limits libarchive’s ability to support some self-extracting archives and ones that have been modified in certain ways.
d372 1 a372 1Archive d393 1 a393 1
mtree d400 1 a400 1 both the NetBSD and FreeBSD versions of mtree(1), although d410 1 a410 1
When reading an d420 2 a421 1
SEE ALSO
d423 17 a439 1ar(1), cpio(1), mkisofs(1), d443 2 a444 3
FreeBSD 8.0 December 27, 2009 FreeBSD 8.0
@ 1.1.1.1 log @Import libarchive 2.8.0: - Infrastructure: - Allow command line tools as fallback for missing compression libraries. If compiled without gzip for example, gunzip will be used automatically. - Improved support for a number of platforms like high-resolution timestamps and Extended Attributes on various Unix systems - New convience interface for creating archives based on disk content, complement of the archive_write_disk interface. - Frontends: - bsdcpio ready for public consumption - hand-written date parser replaces the yacc code - Filter system: - Simplified read filter chains - Option support for filters - LZMA, XZ, uudecode handled - Format support: - Write support for mtree files based on file system or archive content - Basic read support for Joliet - Write support for zip files - Write support for shar archives, both text-only and binary-safe @ text @@ 1.1.1.2 log @libarchive-2.8.2: - Fix NULL deference for short self-extracting zip archives - Don't dereference symlinks on Linux when reading ACLs - Better detection of SHA2 support for old OpenSSL versions - Fix parsing of input files for bsdtar -T - Do not leak setup_xattr into the global namespace - Fix build when an older libarchive is already installed - Use O_BINARY opening files in bsdtar - Include missing archive_crc32.h - Correctly include iconv.h required by libxml2 @ text @d1 375 a375 895 %!PS-Adobe-3.0 %%Creator: groff version 1.19.2 %%CreationDate: Sun Mar 14 02:49:17 2010 %%DocumentNeededResources: font Times-Roman %%DocumentSuppliedResources: procset grops 1.19 2 %%Pages: 61 %%PageOrder: Ascend %%DocumentMedia: Default 612 792 0 () () %%Orientation: Portrait %%EndComments %%BeginDefaults %%PageMedia: Default %%EndDefaults %%BeginProlog %%BeginResource: procset grops 1.19 2 %!PS-Adobe-3.0 Resource-ProcSet /setpacking where{ pop currentpacking true setpacking }if /grops 120 dict dup begin /SC 32 def /A/show load def /B{0 SC 3 -1 roll widthshow}bind def /C{0 exch ashow}bind def /D{0 exch 0 SC 5 2 roll awidthshow}bind def /E{0 rmoveto show}bind def /F{0 rmoveto 0 SC 3 -1 roll widthshow}bind def /G{0 rmoveto 0 exch ashow}bind def /H{0 rmoveto 0 exch 0 SC 5 2 roll awidthshow}bind def /I{0 exch rmoveto show}bind def /J{0 exch rmoveto 0 SC 3 -1 roll widthshow}bind def /K{0 exch rmoveto 0 exch ashow}bind def /L{0 exch rmoveto 0 exch 0 SC 5 2 roll awidthshow}bind def /M{rmoveto show}bind def /N{rmoveto 0 SC 3 -1 roll widthshow}bind def /O{rmoveto 0 exch ashow}bind def /P{rmoveto 0 exch 0 SC 5 2 roll awidthshow}bind def /Q{moveto show}bind def /R{moveto 0 SC 3 -1 roll widthshow}bind def /S{moveto 0 exch ashow}bind def /T{moveto 0 exch 0 SC 5 2 roll awidthshow}bind def /SF{ findfont exch [exch dup 0 exch 0 exch neg 0 0]makefont dup setfont [exch/setfont cvx]cvx bind def }bind def /MF{ findfont [5 2 roll 0 3 1 roll neg 0 0]makefont dup setfont [exch/setfont cvx]cvx bind def }bind def /level0 0 def /RES 0 def /PL 0 def /LS 0 def /MANUAL{ statusdict begin/manualfeed true store end }bind def /PLG{ gsave newpath clippath pathbbox grestore exch pop add exch pop }bind def /BP{ /level0 save def 1 setlinecap 1 setlinejoin 72 RES div dup scale LS{ 90 rotate }{ 0 PL translate }ifelse 1 -1 scale }bind def /EP{ level0 restore showpage }def /DA{ newpath arcn stroke }bind def /SN{ transform .25 sub exch .25 sub exch round .25 add exch round .25 add exch itransform }bind def /DL{ SN moveto SN lineto stroke }bind def /DC{ newpath 0 360 arc closepath }bind def /TM matrix def /DE{ TM currentmatrix pop translate scale newpath 0 0 .5 0 360 arc closepath TM setmatrix }bind def /RC/rcurveto load def /RL/rlineto load def /ST/stroke load def /MT/moveto load def /CL/closepath load def /Fr{ setrgbcolor fill }bind def /setcmykcolor where{ pop /Fk{ setcmykcolor fill }bind def }if /Fg{ setgray fill }bind def /FL/fill load def /LW/setlinewidth load def /Cr/setrgbcolor load def /setcmykcolor where{ pop /Ck/setcmykcolor load def }if /Cg/setgray load def /RE{ findfont dup maxlength 1 index/FontName known not{1 add}if dict begin { 1 index/FID ne{def}{pop pop}ifelse }forall /Encoding exch def dup/FontName exch def currentdict end definefont pop }bind def /DEFS 0 def /EBEGIN{ moveto DEFS begin }bind def /EEND/end load def /CNT 0 def /level1 0 def /PBEGIN{ /level1 save def translate div 3 1 roll div exch scale neg exch neg exch translate 0 setgray 0 setlinecap 1 setlinewidth 0 setlinejoin 10 setmiterlimit []0 setdash /setstrokeadjust where{ pop false setstrokeadjust }if /setoverprint where{ pop false setoverprint }if newpath /CNT countdictstack def userdict begin /showpage{}def /setpagedevice{}def }bind def /PEND{ countdictstack CNT sub{end}repeat level1 restore }bind def end def /setpacking where{ pop setpacking }if %%EndResource %%EndProlog %%BeginSetup %%BeginFeature: *PageSize Default << /PageSize [ 612 792 ] /ImagingBBox null >> setpagedevice %%EndFeature %%IncludeResource: font Times-Roman grops begin/DEFS 1 dict def DEFS begin/u{.001 mul}bind def end/RES 72 def/PL 792 def/LS false def/ENC0[/asciicircum/asciitilde/Scaron/Zcaron /scaron/zcaron/Ydieresis/trademark/quotesingle/Euro/.notdef/.notdef /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/.notdef/space/exclam/quotedbl/numbersign/dollar/percent /ampersand/quoteright/parenleft/parenright/asterisk/plus/comma/hyphen /period/slash/zero/one/two/three/four/five/six/seven/eight/nine/colon /semicolon/less/equal/greater/question/at/A/B/C/D/E/F/G/H/I/J/K/L/M/N/O /P/Q/R/S/T/U/V/W/X/Y/Z/bracketleft/backslash/bracketright/circumflex /underscore/quoteleft/a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y /z/braceleft/bar/braceright/tilde/.notdef/quotesinglbase/guillemotleft /guillemotright/bullet/florin/fraction/perthousand/dagger/daggerdbl /endash/emdash/ff/fi/fl/ffi/ffl/dotlessi/dotlessj/grave/hungarumlaut /dotaccent/breve/caron/ring/ogonek/quotedblleft/quotedblright/oe/lslash /quotedblbase/OE/Lslash/.notdef/exclamdown/cent/sterling/currency/yen /brokenbar/section/dieresis/copyright/ordfeminine/guilsinglleft /logicalnot/minus/registered/macron/degree/plusminus/twosuperior /threesuperior/acute/mu/paragraph/periodcentered/cedilla/onesuperior /ordmasculine/guilsinglright/onequarter/onehalf/threequarters /questiondown/Agrave/Aacute/Acircumflex/Atilde/Adieresis/Aring/AE /Ccedilla/Egrave/Eacute/Ecircumflex/Edieresis/Igrave/Iacute/Icircumflex /Idieresis/Eth/Ntilde/Ograve/Oacute/Ocircumflex/Otilde/Odieresis /multiply/Oslash/Ugrave/Uacute/Ucircumflex/Udieresis/Yacute/Thorn /germandbls/agrave/aacute/acircumflex/atilde/adieresis/aring/ae/ccedilla /egrave/eacute/ecircumflex/edieresis/igrave/iacute/icircumflex/idieresis /eth/ntilde/ograve/oacute/ocircumflex/otilde/odieresis/divide/oslash /ugrave/uacute/ucircumflex/udieresis/yacute/thorn/ydieresis]def /Times-Roman@@0 ENC0/Times-Roman RE %%EndSetup %%Page: 1 1 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF()0 12 Q()0 24 Q()36 48 Q(The follo)0 60 Q (wing pro)-.25 E(vides a brief description of each format supported)-.15 E(by libarchi)0 72 Q -.15(ve)-.25 G 2.5(,w).15 G (ith some information about recognized e)-2.5 E(xtensions or)-.15 E 0 Cg EP %%Page: 7 7 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(limitations of the current library support.)0 12 Q(Note that just because a format is supported by libarchi)0 24 Q .3 -.15(ve d)-.25 H(oes not).15 E(imply that a program that uses libarchi)0 36 Q .3 -.15(ve w)-.25 H(ill support that format.).15 E (Applications that use libarchi)0 48 Q .3 -.15(ve s)-.25 H (pecify which formats the).15 E 2.5(yw)-.15 G(ish)-2.5 E (to support, though man)0 60 Q 2.5(yp)-.15 G(rograms do use libarchi) -2.5 E .3 -.15(ve c)-.25 H(on).15 E -.15(ve)-.4 G(nience).15 E (functions to enable all supported formats.)0 72 Q 0 Cg EP %%Page: 8 8 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(
)36 72 Q 0 Cg EP %%Page: 10 10 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF (All tar formats store each entry in one or more 512-byte records.)0 12 Q(The \214rst record is used for \214le metadata, including \214lename,) 0 24 Q (timestamp, and mode information, and the \214le data is stored in)0 36 Q(subsequent records.)0 48 Q(Later v)0 60 Q(ariants ha)-.25 E .3 -.15 (ve ex)-.2 H(tended this by either appropriating unde\214ned).15 E (areas of the header record, e)0 72 Q (xtending the header to multiple records,)-.15 E 0 Cg EP %%Page: 11 11 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF (or by storing special entries that modify the interpretation of)0 12 Q (subsequent entries.)0 24 Q(
)36 36 Q(
P)0 24 Q(axHeader)-.15 E(directories.)0 36 Q
()36 24 Q(The libarchi)0 36 Q .3 -.15(ve l)-.25 H(ibrary also reads a v).15 E(ariety of commonly-used e) -.25 E(xtensions to)-.15 E(the basic tar format.)0 48 Q(These e)0 60 Q (xtensions are recognized automatically whene)-.15 E -.15(ve)-.25 G 2.5 (rt).15 G(he)-2.5 E 2.5(ya)-.15 G(ppear)-2.5 E(.)-.55 E(
)36 48 Q(The \214rst tar program appeared in Se)0 60 Q -.15(ve)-.25 G(nth Edition Unix in 1979.).15 E(The \214rst of)0 72 Q(\214cial standard for the tar \214le format w)-.25 E(as the)-.1 E 0 Cg EP %%Page: 27 27 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF -.74(``)0 12 S(ustar').74 E(')-.74 E (\(Unix Standard T)0 24 Q(ar\) format de\214ned by POSIX in 1988.)-.8 E (POSIX.1-2001 e)0 36 Q(xtended the ustar format to create the)-.15 E -.74(``)0 48 S(pax interchange').74 E(')-.74 E(format.)0 60 Q (
)36 24 Q (Cpio \214rst appeared in PWB/UNIX 1.0, which w)0 36 Q (as released within)-.1 E -1.11(AT)0 48 S(&T in 1977.)1.11 E (PWB/UNIX 1.0 formed the basis of System III Unix, released outside)0 60 Q(of A)0 72 Q(T&T in 1981.)-1.11 E 0 Cg EP %%Page: 36 36 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(This mak)0 12 Q(es cpio older than tar)-.1 E 2.5 (,a)-.4 G(lthough cpio w)-2.5 E(as not included)-.1 E(in V)0 24 Q (ersion 7 A)-1.11 E(T&T Unix.)-1.11 E (As a result, the tar command became much better kno)0 36 Q(wn in uni) -.25 E -.15(ve)-.25 G(rsities).15 E(and research groups that used V)0 48 Q(ersion 7.)-1.11 E(The combination of the)0 60 Q(\214nd)0 72 Q 0 Cg EP %%Page: 37 37 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(and)0 12 Q(cpio)0 24 Q(utilities pro)0 36 Q(vided v)-.15 E(ery precise control o)-.15 E -.15(ve)-.15 G 2.5<728c> .15 G(le selection.)-2.5 E(Unfortunately)0 48 Q 2.5(,t)-.65 G (he format has man)-2.5 E 2.5(yl)-.15 G(imitations that mak)-2.5 E 2.5 (ei)-.1 G 2.5(tu)-2.5 G(nsuitable)-2.5 E(for widespread use.)0 60 Q (Only the POSIX format permits \214les o)0 72 Q -.15(ve)-.15 G 2.5(r4) .15 G(GB, and its 18-bit)-2.5 E 0 Cg EP %%Page: 38 38 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(limit for most other \214elds mak)0 12 Q (es it unsuitable for modern systems.)-.1 E (In addition, cpio formats only store numeric UID/GID v)0 24 Q (alues \(not)-.25 E(usernames and group names\), which can mak)0 36 Q 2.5(ei)-.1 G 2.5(tv)-2.5 G(ery dif)-2.65 E(\214cult to correctly)-.25 E (transfer archi)0 48 Q -.15(ve)-.25 G 2.5(sa).15 G (cross systems with dissimilar user numbering.)-2.5 E(
) 36 48 Q(When reading an mtree \214le, libarchi)0 60 Q .3 -.15(ve w)-.25 H(ill locate the corresponding).15 E(\214les on disk using the)0 72 Q 0 Cg EP %%Page: 58 58 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(contents)0 12 Q -.1(ke)0 24 S(yw) -.05 E(ord if present or the re)-.1 E(gular \214lename.)-.15 E (If it can locate and open the \214le on disk, it will use that)0 36 Q (to \214ll in an)0 48 Q 2.5(ym)-.15 G (etadata that is missing from the mtree \214le)-2.5 E (and will read the \214le contents and return those to the program)0 60 Q(using libarchi)0 72 Q -.15(ve)-.25 G(.).15 E 0 Cg EP %%Page: 59 59 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF (If it cannot locate and open the \214le on disk, libarchi)0 12 Q -.15 (ve)-.25 G(will return an error for an)0 24 Q 2.5(ya)-.15 G (ttempt to read the entry)-2.5 E(body)0 36 Q(.)-.65 E (
libarchive-formats(5) FreeBSD File Formats Manual libarchive-formats(5)
NAME
libarchive-formats — archive formats supported by the libarchive library
DESCRIPTION
The libarchive(3) library reads and writes a variety of streaming archive formats. Generally speaking, all of these archive formats consist of a series of ‘‘entries’’. Each entry stores a single file system object, such as a file, directory, or symbolic link.
The following provides a brief description of each format supported by libarchive, with some information about recognized extensions or limitations of the current library support. Note that just because a format is supported by libarchive does not imply that a program that uses libarchive will support that format. Applications that use libarchive specify which formats they wish to support, though many programs do use libarchive convenience functions to enable all supported formats.
Tar
Formats
The libarchive(3) library can read most tar archives.
However, it only writes POSIX-standard
‘‘ustar’’ and ‘‘pax
interchange’’ formats.
All tar formats store each entry in one or more 512-byte records. The first record is used for file metadata, including filename, timestamp, and mode information, and the file data is stored in subsequent records. Later variants have extended this by either appropriating undefined areas of the header record, extending the header to multiple records, or by storing special entries that modify the interpretation of subsequent entries.
gnutar
The libarchive(3) library can read GNU-format tar archives. It currently supports the most popular GNU extensions, including modern long filename and linkname support, as well as atime and ctime data. The libarchive library does not support multi-volume archives, nor the old GNU long filename format. It can read GNU sparse file entries, including the new POSIX-based formats, but cannot write GNU sparse file entries.
pax
The libarchive(3) library can read and write POSIX-compliant pax interchange format archives. Pax interchange format archives are an extension of the older ustar format that adds a separate entry with additional attributes stored as key/value pairs immediately before each regular entry. The presence of these additional entries is the only difference between pax interchange format and the older ustar format. The extended attributes are of unlimited length and are stored as UTF-8 Unicode strings. Keywords defined in the standard are in all lowercase; vendors are allowed to define custom keys by preceding them with the vendor name in all uppercase. When writing pax archives, libarchive uses many of the SCHILY keys defined by Joerg Schilling’s ‘‘star’’ archiver and a few LIBARCHIVE keys. The libarchive library can read most of the SCHILY keys and most of the GNU keys introduced by GNU tar. It silently ignores any keywords that it does not understand.
restricted pax
The libarchive library can also write pax archives in which it attempts to suppress the extended attributes entry whenever possible. The result will be identical to a ustar archive unless the extended attributes entry is required to store a long file name, long linkname, extended ACL, file flags, or if any of the standard ustar data (user name, group name, UID, GID, etc) cannot be fully represented in the ustar header. In all cases, the result can be dearchived by any program that can read POSIX-compliant pax interchange format archives. Programs that correctly read ustar format (see below) will also be able to read this format; any extended attributes will be extracted as separate files stored in PaxHeader directories.
ustar
The libarchive library can both read and write this format. This format has the following limitations:
•
Device major and minor numbers are limited to 21 bits. Nodes with larger numbers will not be added to the archive.
•
Path names in the archive are limited to 255 bytes. (Shorter if there is no / character in exactly the right place.)
•
Symbolic links and hard links are stored in the archive with the name of the referenced file. This name is limited to 100 bytes.
•
Extended attributes, file flags, and other extended security information cannot be stored.
•
Archive entries are limited to 8 gigabytes in size.
Note that the pax interchange format has none of these restrictions.
The libarchive library also reads a variety of commonly-used extensions to the basic tar format. These extensions are recognized automatically whenever they appear.
Numeric extensions.
The POSIX standards require fixed-length numeric fields to be written with some character position reserved for terminators. Libarchive allows these fields to be written without terminator characters. This extends the allowable range; in particular, ustar archives with this extension can support entries up to 64 gigabytes in size. Libarchive also recognizes base-256 values in most numeric fields. This essentially removes all limitations on file size, modification time, and device numbers.
Solaris extensions
Libarchive recognizes ACL and extended attribute records written by Solaris tar. Currently, libarchive only has support for old-style ACLs; the newer NFSv4 ACLs are recognized but discarded.
The first tar program appeared in Seventh Edition Unix in 1979. The first official standard for the tar file format was the ‘‘ustar’’ (Unix Standard Tar) format defined by POSIX in 1988. POSIX.1-2001 extended the ustar format to create the ‘‘pax interchange’’ format.
Cpio
Formats
The libarchive library can read a number of common cpio
variants and can write ‘‘odc’’ and
‘‘newc’’ format archives. A cpio
archive stores each entry as a fixed-size header followed by
a variable-length filename and variable-length data. Unlike
the tar format, the cpio format does only minimal padding of
the header or file data. There are several cpio variants,
which differ primarily in how they store the initial header:
some store the values as octal or hexadecimal numbers in
ASCII, others as binary values of varying byte order and
length.
binary
The libarchive library transparently reads both big-endian and little-endian variants of the original binary cpio format. This format used 32-bit binary values for file size and mtime, and 16-bit binary values for the other fields.
odc
The libarchive library can both read and write this POSIX-standard format, which is officially known as the ‘‘cpio interchange format’’ or the ‘‘octet-oriented cpio archive format’’ and sometimes unofficially referred to as the ‘‘old character format’’. This format stores the header contents as octal values in ASCII. It is standard, portable, and immune from byte-order confusion. File sizes and mtime are limited to 33 bits (8GB file size), other fields are limited to 18 bits.
SVR4
The libarchive library can read both CRC and non-CRC variants of this format. The SVR4 format uses eight-digit hexadecimal values for all header fields. This limits file size to 4GB, and also limits the mtime and other fields to 32 bits. The SVR4 format can optionally include a CRC of the file contents, although libarchive does not currently verify this CRC.
Cpio first appeared in PWB/UNIX 1.0, which was released within AT&T in 1977. PWB/UNIX 1.0 formed the basis of System III Unix, released outside of AT&T in 1981. This makes cpio older than tar, although cpio was not included in Version 7 AT&T Unix. As a result, the tar command became much better known in universities and research groups that used Version 7. The combination of the find and cpio utilities provided very precise control over file selection. Unfortunately, the format has many limitations that make it unsuitable for widespread use. Only the POSIX format permits files over 4GB, and its 18-bit limit for most other fields makes it unsuitable for modern systems. In addition, cpio formats only store numeric UID/GID values (not usernames and group names), which can make it very difficult to correctly transfer archives across systems with dissimilar user numbering.
Shar
Formats
A ‘‘shell archive’’ is a shell
script that, when executed on a POSIX-compliant system, will
recreate a collection of file system objects. The libarchive
library can write two different kinds of shar archives:
shar
The traditional shar format uses a limited set of POSIX commands, including echo(1), mkdir(1), and sed(1). It is suitable for portably archiving small collections of plain text files. However, it is not generally well-suited for large archives (many implementations of sh(1) have limits on the size of a script) nor should it be used with non-text files.
shardump
This format is similar to shar but encodes files using uuencode(1) so that the result will be a plain text file regardless of the file contents. It also includes additional shell commands that attempt to reproduce as many file attributes as possible, including owner, mode, and flags. The additional commands used to restore file attributes make shardump archives less portable than plain shar archives.
ISO9660
format
Libarchive can read and extract from files containing
ISO9660-compliant CDROM images. In many cases, this can
remove the need to burn a physical CDROM just in order to
read the files contained in an ISO9660 image. It also avoids
security and complexity issues that come with virtual mounts
and loopback devices. Libarchive supports the most common
Rockridge extensions and has partial support for Joliet
extensions. If both extensions are present, the Joliet
extensions will be used and the Rockridge extensions will be
ignored. In particular, this can create problems with
hardlinks and symlinks, which are supported by Rockridge but
not by Joliet.
Zip
format
Libarchive can read and write zip format archives that have
uncompressed entries and entries compressed with the
‘‘deflate’’ algorithm. Older zip
compression algorithms are not supported. It can extract jar
archives, archives that use Zip64 extensions and many
self-extracting zip archives. Libarchive reads Zip archives
as they are being streamed, which allows it to read archives
of arbitrary size. It currently does not use the central
directory; this limits libarchive’s ability to support
some self-extracting archives and ones that have been
modified in certain ways.
Archive
(library) file format
The Unix archive format (commonly created by the ar(1)
archiver) is a general-purpose format which is used almost
exclusively for object files to be read by the link editor
ld(1). The ar format has never been standardised. There are
two common variants: the GNU format derived from SVR4, and
the BSD format, which first appeared in 4.4BSD. The two
differ primarily in their handling of filenames longer than
15 characters: the GNU/SVR4 variant writes a filename table
at the beginning of the archive; the BSD format stores each
long filename in an extension area adjacent to the entry.
Libarchive can read both extensions, including archives that
may include both types of long filenames. Programs using
libarchive can write GNU/SVR4 format if they provide a
filename table to be written into the archive before any of
the entries. Any entries whose names are not in the filename
table will be written using BSD-style long filenames. This
can cause problems for programs such as GNU ld that do not
support the BSD-style long filenames.
mtree
Libarchive can read and write files in mtree(5) format. This
format is not a true archive format, but rather a textual
description of a file hierarchy in which each line specifies
the name of a file and provides specific metadata about that
file. Libarchive can read all of the keywords supported by
both the NetBSD and FreeBSD versions of mtree(1), although
many of the keywords cannot currently be stored in an
archive_entry object. When writing, libarchive supports use
of the archive_write_set_options(3) interface to specify
which keywords should be included in the output. If
libarchive was compiled with access to suitable
cryptographic libraries (such as the OpenSSL libraries), it
can compute hash entries such as sha512 or md5
from file data being written to the mtree writer.
When reading an mtree file, libarchive will locate the corresponding files on disk using the contents keyword if present or the regular filename. If it can locate and open the file on disk, it will use that to fill in any metadata that is missing from the mtree file and will read the file contents and return those to the program using libarchive. If it cannot locate and open the file on disk, libarchive will return an error for any attempt to read the entry body.
SEE ALSO
ar(1), cpio(1), mkisofs(1), shar(1), tar(1), zip(1), zlib(3), cpio(5), mtree(5), tar(5)
FreeBSD 9.0 December 27, 2009 FreeBSD 9.0
)36 48 Q(The follo)0 60 Q (wing pro)-.25 E(vides a brief description of each format supported)-.15 E(by libarchi)0 72 Q -.15(ve)-.25 G 2.5(,w).15 G (ith some information about recognized e)-2.5 E(xtensions or)-.15 E 0 Cg EP %%Page: 7 7 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(limitations of the current library support.)0 12 Q(Note that just because a format is supported by libarchi)0 24 Q .3 -.15(ve d)-.25 H(oes not).15 E(imply that a program that uses libarchi)0 36 Q .3 -.15(ve w)-.25 H(ill support that format.).15 E (Applications that use libarchi)0 48 Q .3 -.15(ve s)-.25 H (pecify which formats the).15 E 2.5(yw)-.15 G(ish)-2.5 E (to support, though man)0 60 Q 2.5(yp)-.15 G(rograms do use libarchi) -2.5 E .3 -.15(ve c)-.25 H(on).15 E -.15(ve)-.4 G(nience).15 E (functions to enable all supported formats.)0 72 Q 0 Cg EP %%Page: 8 8 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(
)36 72 Q 0 Cg EP %%Page: 10 10 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF (All tar formats store each entry in one or more 512-byte records.)0 12 Q(The \214rst record is used for \214le metadata, including \214lename,) 0 24 Q (timestamp, and mode information, and the \214le data is stored in)0 36 Q(subsequent records.)0 48 Q(Later v)0 60 Q(ariants ha)-.25 E .3 -.15 (ve ex)-.2 H(tended this by either appropriating unde\214ned).15 E (areas of the header record, e)0 72 Q (xtending the header to multiple records,)-.15 E 0 Cg EP %%Page: 11 11 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF (or by storing special entries that modify the interpretation of)0 12 Q (subsequent entries.)0 24 Q(
)36 36 Q(
P)0 24 Q(axHeader)-.15 E(directories.)0 36 Q
()36 24 Q(The libarchi)0 36 Q .3 -.15(ve l)-.25 H(ibrary also reads a v).15 E(ariety of commonly-used e) -.25 E(xtensions to)-.15 E(the basic tar format.)0 48 Q(These e)0 60 Q (xtensions are recognized automatically whene)-.15 E -.15(ve)-.25 G 2.5 (rt).15 G(he)-2.5 E 2.5(ya)-.15 G(ppear)-2.5 E(.)-.55 E(
)36 48 Q(The \214rst tar program appeared in Se)0 60 Q -.15(ve)-.25 G(nth Edition Unix in 1979.).15 E(The \214rst of)0 72 Q(\214cial standard for the tar \214le format w)-.25 E(as the)-.1 E 0 Cg EP %%Page: 27 27 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF -.74(``)0 12 S(ustar').74 E(')-.74 E (\(Unix Standard T)0 24 Q(ar\) format de\214ned by POSIX in 1988.)-.8 E (POSIX.1-2001 e)0 36 Q(xtended the ustar format to create the)-.15 E -.74(``)0 48 S(pax interchange').74 E(')-.74 E(format.)0 60 Q (
)36 24 Q (Cpio \214rst appeared in PWB/UNIX 1.0, which w)0 36 Q (as released within)-.1 E -1.11(AT)0 48 S(&T in 1977.)1.11 E (PWB/UNIX 1.0 formed the basis of System III Unix, released outside)0 60 Q(of A)0 72 Q(T&T in 1981.)-1.11 E 0 Cg EP %%Page: 36 36 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(This mak)0 12 Q(es cpio older than tar)-.1 E 2.5 (,a)-.4 G(lthough cpio w)-2.5 E(as not included)-.1 E(in V)0 24 Q (ersion 7 A)-1.11 E(T&T Unix.)-1.11 E (As a result, the tar command became much better kno)0 36 Q(wn in uni) -.25 E -.15(ve)-.25 G(rsities).15 E(and research groups that used V)0 48 Q(ersion 7.)-1.11 E(The combination of the)0 60 Q(\214nd)0 72 Q 0 Cg EP %%Page: 37 37 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(and)0 12 Q(cpio)0 24 Q(utilities pro)0 36 Q(vided v)-.15 E(ery precise control o)-.15 E -.15(ve)-.15 G 2.5<728c> .15 G(le selection.)-2.5 E(Unfortunately)0 48 Q 2.5(,t)-.65 G (he format has man)-2.5 E 2.5(yl)-.15 G(imitations that mak)-2.5 E 2.5 (ei)-.1 G 2.5(tu)-2.5 G(nsuitable)-2.5 E(for widespread use.)0 60 Q (Only the POSIX format permits \214les o)0 72 Q -.15(ve)-.15 G 2.5(r4) .15 G(GB, and its 18-bit)-2.5 E 0 Cg EP %%Page: 38 38 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(limit for most other \214elds mak)0 12 Q (es it unsuitable for modern systems.)-.1 E (In addition, cpio formats only store numeric UID/GID v)0 24 Q (alues \(not)-.25 E(usernames and group names\), which can mak)0 36 Q 2.5(ei)-.1 G 2.5(tv)-2.5 G(ery dif)-2.65 E(\214cult to correctly)-.25 E (transfer archi)0 48 Q -.15(ve)-.25 G 2.5(sa).15 G (cross systems with dissimilar user numbering.)-2.5 E(
) 36 48 Q(When reading an mtree \214le, libarchi)0 60 Q .3 -.15(ve w)-.25 H(ill locate the corresponding).15 E(\214les on disk using the)0 72 Q 0 Cg EP %%Page: 58 58 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF(contents)0 12 Q -.1(ke)0 24 S(yw) -.05 E(ord if present or the re)-.1 E(gular \214lename.)-.15 E (If it can locate and open the \214le on disk, it will use that)0 36 Q (to \214ll in an)0 48 Q 2.5(ym)-.15 G (etadata that is missing from the mtree \214le)-2.5 E (and will read the \214le contents and return those to the program)0 60 Q(using libarchi)0 72 Q -.15(ve)-.25 G(.).15 E 0 Cg EP %%Page: 59 59 %%BeginPageSetup BP %%EndPageSetup /F0 10/Times-Roman@@0 SF (If it cannot locate and open the \214le on disk, libarchi)0 12 Q -.15 (ve)-.25 G(will return an error for an)0 24 Q 2.5(ya)-.15 G (ttempt to read the entry)-2.5 E(body)0 36 Q(.)-.65 E (
LIBARCHIVE-FORMATS(5) BSD File Formats Manual LIBARCHIVE-FORMATS(5)
NAME
libarchive-formats — archive formats supported by the libarchive library
DESCRIPTION
The libarchive(3) library reads and writes a variety of streaming archive formats. Generally speaking, all of these archive formats consist of a series of ’’entries’’. Each entry stores a single file system object, such as a file, directory, or symbolic link.
The following provides a brief description of each format supported by libarchive, with some information about recognized extensions or limitations of the current library support. Note that just because a format is supported by libarchive does not imply that a program that uses libarchive will support that format. Applications that use libarchive specify which formats they wish to support, though many programs do use libarchive convenience functions to enable all supported formats.
Tar
Formats
The libarchive(3) library can read most tar archives. It can
write POSIX-standard ’’ustar’’ and
’’pax interchange’’ formats as well
as v7 tar format and a subset of the legacy GNU tar
format.
All tar formats store each entry in one or more 512-byte records. The first record is used for file metadata, including filename, timestamp, and mode information, and the file data is stored in subsequent records. Later variants have extended this by either appropriating undefined areas of the header record, extending the header to multiple records, or by storing special entries that modify the interpretation of subsequent entries.
gnutar
The libarchive(3) library can read most GNU-format tar archives. It currently supports the most popular GNU extensions, including modern long filename and linkname support, as well as atime and ctime data. The libarchive library does not support multi-volume archives, nor the old GNU long filename format. It can read GNU sparse file entries, including the new POSIX-based formats.
The libarchive(3) library can write GNU tar format, including long filename and linkname support, as well as atime and ctime data.
pax
The libarchive(3) library can read and write POSIX-compliant pax interchange format archives. Pax interchange format archives are an extension of the older ustar format that adds a separate entry with additional attributes stored as key/value pairs immediately before each regular entry. The presence of these additional entries is the only difference between pax interchange format and the older ustar format. The extended attributes are of unlimited length and are stored as UTF-8 Unicode strings. Keywords defined in the standard are in all lowercase; vendors are allowed to define custom keys by preceding them with the vendor name in all uppercase. When writing pax archives, libarchive uses many of the SCHILY keys defined by Joerg Schilling’s ’’star’’ archiver and a few LIBARCHIVE keys. The libarchive library can read most of the SCHILY keys and most of the GNU keys introduced by GNU tar. It silently ignores any keywords that it does not understand.
The pax interchange format converts filenames to Unicode and stores them using the UTF-8 encoding. Prior to libarchive 3.0, libarchive erroneously assumed that the system wide-character routines natively supported Unicode. This caused it to mis-handle non-ASCII filenames on systems that did not satisfy this assumption.
restricted pax
The libarchive library can also write pax archives in which it attempts to suppress the extended attributes entry whenever possible. The result will be identical to a ustar archive unless the extended attributes entry is required to store a long file name, long linkname, extended ACL, file flags, or if any of the standard ustar data (user name, group name, UID, GID, etc) cannot be fully represented in the ustar header. In all cases, the result can be dearchived by any program that can read POSIX-compliant pax interchange format archives. Programs that correctly read ustar format (see below) will also be able to read this format; any extended attributes will be extracted as separate files stored in PaxHeader directories.
ustar
The libarchive library can both read and write this format. This format has the following limitations:
•
Device major and minor numbers are limited to 21 bits. Nodes with larger numbers will not be added to the archive.
•
Path names in the archive are limited to 255 bytes. (Shorter if there is no / character in exactly the right place.)
•
Symbolic links and hard links are stored in the archive with the name of the referenced file. This name is limited to 100 bytes.
•
Extended attributes, file flags, and other extended security information cannot be stored.
•
Archive entries are limited to 8 gigabytes in size.
Note that the pax interchange format has none of these restrictions. The ustar format is old and widely supported. It is recommended when compatibility is the primary concern.
v7
The libarchive library can read and write the legacy v7 tar format. This format has the following limitations:
•
Only regular files, directories, and symbolic links can be archived. Block and character device nodes, FIFOs, and sockets cannot be archived.
•
Path names in the archive are limited to 100 bytes.
•
Symbolic links and hard links are stored in the archive with the name of the referenced file. This name is limited to 100 bytes.
•
User and group information are stored as numeric IDs; there is no provision for storing user or group names.
•
Extended attributes, file flags, and other extended security information cannot be stored.
•
Archive entries are limited to 8 gigabytes in size.
Generally, users should prefer the ustar format for portability as the v7 tar format is both less useful and less portable.
The libarchive library also reads a variety of commonly-used extensions to the basic tar format. These extensions are recognized automatically whenever they appear.
Numeric extensions.
The POSIX standards require fixed-length numeric fields to be written with some character position reserved for terminators. Libarchive allows these fields to be written without terminator characters. This extends the allowable range; in particular, ustar archives with this extension can support entries up to 64 gigabytes in size. Libarchive also recognizes base-256 values in most numeric fields. This essentially removes all limitations on file size, modification time, and device numbers.
Solaris extensions
Libarchive recognizes ACL and extended attribute records written by Solaris tar. Currently, libarchive only has support for old-style ACLs; the newer NFSv4 ACLs are recognized but discarded.
The first tar program appeared in Seventh Edition Unix in 1979. The first official standard for the tar file format was the ’’ustar’’ (Unix Standard Tar) format defined by POSIX in 1988. POSIX.1-2001 extended the ustar format to create the ’’pax interchange’’ format.
Cpio
Formats
The libarchive library can read a number of common cpio
variants and can write ’’odc’’ and
’’newc’’ format archives. A cpio
archive stores each entry as a fixed-size header followed by
a variable-length filename and variable-length data. Unlike
the tar format, the cpio format does only minimal padding of
the header or file data. There are several cpio variants,
which differ primarily in how they store the initial header:
some store the values as octal or hexadecimal numbers in
ASCII, others as binary values of varying byte order and
length.
binary
The libarchive library transparently reads both big-endian and little-endian variants of the original binary cpio format. This format used 32-bit binary values for file size and mtime, and 16-bit binary values for the other fields.
odc
The libarchive library can both read and write this POSIX-standard format, which is officially known as the ’’cpio interchange format’’ or the ’’octet-oriented cpio archive format’’ and sometimes unofficially referred to as the ’’old character format’’. This format stores the header contents as octal values in ASCII. It is standard, portable, and immune from byte-order confusion. File sizes and mtime are limited to 33 bits (8GB file size), other fields are limited to 18 bits.
SVR4/newc
The libarchive library can read both CRC and non-CRC variants of this format. The SVR4 format uses eight-digit hexadecimal values for all header fields. This limits file size to 4GB, and also limits the mtime and other fields to 32 bits. The SVR4 format can optionally include a CRC of the file contents, although libarchive does not currently verify this CRC.
Cpio first appeared in PWB/UNIX 1.0, which was released within AT&T in 1977. PWB/UNIX 1.0 formed the basis of System III Unix, released outside of AT&T in 1981. This makes cpio older than tar, although cpio was not included in Version 7 AT&T Unix. As a result, the tar command became much better known in universities and research groups that used Version 7. The combination of the find and cpio utilities provided very precise control over file selection. Unfortunately, the format has many limitations that make it unsuitable for widespread use. Only the POSIX format permits files over 4GB, and its 18-bit limit for most other fields makes it unsuitable for modern systems. In addition, cpio formats only store numeric UID/GID values (not usernames and group names), which can make it very difficult to correctly transfer archives across systems with dissimilar user numbering.
Shar
Formats
A ’’shell archive’’ is a shell
script that, when executed on a POSIX-compliant system, will
recreate a collection of file system objects. The libarchive
library can write two different kinds of shar archives:
shar
The traditional shar format uses a limited set of POSIX commands, including echo(1), mkdir(1), and sed(1). It is suitable for portably archiving small collections of plain text files. However, it is not generally well-suited for large archives (many implementations of sh(1) have limits on the size of a script) nor should it be used with non-text files.
shardump
This format is similar to shar but encodes files using uuencode(1) so that the result will be a plain text file regardless of the file contents. It also includes additional shell commands that attempt to reproduce as many file attributes as possible, including owner, mode, and flags. The additional commands used to restore file attributes make shardump archives less portable than plain shar archives.
ISO9660
format
Libarchive can read and extract from files containing
ISO9660-compliant CDROM images. In many cases, this can
remove the need to burn a physical CDROM just in order to
read the files contained in an ISO9660 image. It also avoids
security and complexity issues that come with virtual mounts
and loopback devices. Libarchive supports the most common
Rockridge extensions and has partial support for Joliet
extensions. If both extensions are present, the Joliet
extensions will be used and the Rockridge extensions will be
ignored. In particular, this can create problems with
hardlinks and symlinks, which are supported by Rockridge but
not by Joliet.
Libarchive reads ISO9660 images using a streaming strategy. This allows it to read compressed images directly (decompressing on the fly) and allows it to read images directly from network sockets, pipes, and other non-seekable data sources. This strategy works well for optimized ISO9660 images created by many popular programs. Such programs collect all directory information at the beginning of the ISO9660 image so it can be read from a physical disk with a minimum of seeking. However, not all ISO9660 images can be read in this fashion.
Libarchive can also write ISO9660 images. Such images are fully optimized with the directory information preceding all file data. This is done by storing all file data to a temporary file while collecting directory information in memory. When the image is finished, libarchive writes out the directory structure followed by the file data. The location used for the temporary file can be changed by the usual environment variables.
Zip
format
Libarchive can read and write zip format archives that have
uncompressed entries and entries compressed with the
’’deflate’’ algorithm. Other zip
compression algorithms are not supported. It can extract jar
archives, archives that use Zip64 extensions and
self-extracting zip archives. Libarchive can use either of
two different strategies for reading Zip archives: a
streaming strategy which is fast and can handle extremely
large archives, and a seeking strategy which can correctly
process self-extracting Zip archives and archives with
deleted members or other in-place modifications.
The streaming reader processes Zip archives as they are read. It can read archives of arbitrary size from tape or network sockets, and can decode Zip archives that have been separately compressed or encoded. However, self-extracting Zip archives and archives with certain types of modifications cannot be correctly handled. Such archives require that the reader first process the Central Directory, which is ordinarily located at the end of a Zip archive and is thus inaccessible to the streaming reader. If the program using libarchive has enabled seek support, then libarchive will use this to processes the central directory first.
In particular, the seeking reader must be used to correctly handle self-extracting archives. Such archives consist of a program followed by a regular Zip archive. The streaming reader cannot parse the initial program portion, but the seeking reader starts by reading the Central Directory from the end of the archive. Similarly, Zip archives that have been modified in-place can have deleted entries or other garbage data that can only be accurately detected by first reading the Central Directory.
Archive
(library) file format
The Unix archive format (commonly created by the ar(1)
archiver) is a general-purpose format which is used almost
exclusively for object files to be read by the link editor
ld(1). The ar format has never been standardised. There are
two common variants: the GNU format derived from SVR4, and
the BSD format, which first appeared in 4.4BSD. The two
differ primarily in their handling of filenames longer than
15 characters: the GNU/SVR4 variant writes a filename table
at the beginning of the archive; the BSD format stores each
long filename in an extension area adjacent to the entry.
Libarchive can read both extensions, including archives that
may include both types of long filenames. Programs using
libarchive can write GNU/SVR4 format if they provide an
entry called // containing a filename table to be
written into the archive before any of the entries. Any
entries whose names are not in the filename table will be
written using BSD-style long filenames. This can cause
problems for programs such as GNU ld that do not support the
BSD-style long filenames.
mtree
Libarchive can read and write files in mtree(5) format. This
format is not a true archive format, but rather a textual
description of a file hierarchy in which each line specifies
the name of a file and provides specific metadata about that
file. Libarchive can read all of the keywords supported by
both the NetBSD and FreeBSD versions of mtree(8), although
many of the keywords cannot currently be stored in an
archive_entry object. When writing, libarchive supports use
of the archive_write_set_options(3) interface to specify
which keywords should be included in the output. If
libarchive was compiled with access to suitable
cryptographic libraries (such as the OpenSSL libraries), it
can compute hash entries such as sha512 or md5
from file data being written to the mtree writer.
When reading an mtree file, libarchive will locate the corresponding files on disk using the contents keyword if present or the regular filename. If it can locate and open the file on disk, it will use that to fill in any metadata that is missing from the mtree file and will read the file contents and return those to the program using libarchive. If it cannot locate and open the file on disk, libarchive will return an error for any attempt to read the entry body.
7-Zip
Libarchive can read and write 7-Zip format archives. TODO:
Need more information
CAB
Libarchive can read Microsoft Cabinet (
’’CAB’’) format archives. TODO: Need
more information.
LHA
TODO: Information about libarchive’s LHA support
RAR
Libarchive has limited support for reading RAR format
archives. Currently, libarchive can read RARv3 format
archives which have been either created uncompressed, or
compressed using any of the compression methods supported by
the RARv3 format. Libarchive can also read self-extracting
RAR archives.
Warc
Libarchive can read and write ’’web
archives’’. TODO: Need more information
XAR
Libarchive can read and write the XAR format used by many
Apple tools. TODO: Need more information
SEE ALSO
ar(1), cpio(1), mkisofs(1), shar(1), tar(1), zip(1), zlib(3), cpio(5), mtree(5), tar(5)
BSD March 18, 2012 BSD
This is the POSIX standardized format, which is officially known as the “cpio interchange format” or the “octet-oriented cpio archive format” and sometimes unofficially referred to as the “old character format”. This format stores the header contents as octal values in ASCII. It is standard, portable, and immune from byte-order confusion. File sizes and mtime are limited to 33 bits (8GB file size), other fields are limited to 18 bits.
@ 1.1.1.11 log @libarchive: import version 3.7.3 @ text @d2 1 a2 1 @ 1.1.1.12 log @Libarchive 3.7.4 is a bugfix and security release Security fixes: rar: Fix OOB in rar e8 filter (CVE-2024-26256) zip: Fix out of boundary access Important bugfixes: 7zip: Limit amount of properties bsdtar: Fix error handling around strtol() usages passphrase: Improve newline handling on Windows passphrase: Never allow empty passwords rar: Fix "File CRC Error" when extracting specific rar4 archives xar: Avoid infinite link loop zip: Update AppleDouble support for directories zstd: Implement core detection @ text @d2 1 a2 1 @ 1.1.1.13 log @libarchive: imported version 3.7.5 Libarchive 3.7.5 Security fixes: fix multiple vulnerabilities identified by SAST cpio: ignore out-of-range gid/uid/size/ino and harden AFIO parsing lzop: prevent integer overflow rar4: protect copy_from_lzss_window_to_unp() rar4: fix CVE-2024-26256 rar4: fix OOB in delta and audio filter rar4: fix out of boundary access with large files rar4: add boundary checks to rgb filter rar4: fix OOB access with unicode filenames rar5: clear 'data ready' cache on window buffer reallocs rpm: calculate huge header sizes correctly unzip: unify EOF handling util: fix out of boundary access in mktemp functions uu: stop processing if lines are too long Important bugfixes: 7zip: fix issue when skipping first file in 7zip archive that is a multiple of 65536 bytes ar: fix archive entries having no type lha: do not allow negative file sizes lha: fix integer truncation on 32-bit systems shar: check strdup return value rar5: don't try to read rediculously long names xar: fix another infinite loop and expat error handling many Windows fixes, cleanups and improvements @ text @d2 1 a2 1 @ 1.1.1.14 log @libarchove: import version 3.7.7 @ text @d1 2 a2 2 d23 2 a24 2LIBARCHIVE-FORMATS(5) File Formats Manual LIBARCHIVE-FORMATS(5)
d28 3 a30 2libarchive-formats — archive formats supported by the libarchive library
d34 6 a39 6The libarchive(3) library reads and writes a variety of streaming archive formats. Generally speaking, all of these archive formats consist of a series of “entries”. Each entry stores a single file system object, such as a file, directory, or symbolic link.
d41 1 a41 1The following d52 6 a57 2
Tar Formats
d59 1 a59 7The libarchive(3) library can read most tar archives. It can write POSIX-standard “ustar” and “pax interchange” formats as well as v7 tar format and a subset of the legacy GNU tar format.
All tar formats d71 13 a83 13
The libarchive(3) library can read most GNU-format tar archives. It currently supports the most popular GNU extensions, including modern long filename and linkname support, as well as atime and ctime data. The libarchive library does not support multi-volume archives, nor the old GNU long filename format. It can read GNU sparse file entries, including the new POSIX-based formats.
The libarchive(3) library can write GNU tar format, including long filename and linkname support, as well as atime and ctime data.
d87 18 a104 19The libarchive(3) library can read and write POSIX-compliant pax interchange format archives. Pax interchange format archives are an extension of the older ustar format that adds a separate entry with additional attributes stored as key/value pairs immediately before each regular entry. The presence of these additional entries is the only difference between pax interchange format and the older ustar format. The extended attributes are of unlimited length and are stored as UTF-8 Unicode strings. Keywords defined in the standard are in all lowercase; vendors are allowed to define custom keys by preceding them with the vendor name in all uppercase. When writing pax archives, libarchive uses many of the SCHILY keys defined by Joerg Schilling’s “star” archiver and a few LIBARCHIVE keys. The libarchive library can read most of the SCHILY keys and most of the GNU keys introduced by GNU tar. It silently ignores any keywords that it does not understand.
d106 1 a106 1The pax d116 1 a116 1
The libarchive library can also d133 1 a133 1
The libarchive d139 1 a139 1
Device major and minor numbers d145 1 a145 1
Path names in the archive are d151 1 a151 1
Symbolic links and hard links d157 1 a157 1
Extended attributes, file d163 1 a163 1
Archive entries are limited to d166 1 a166 1
Note that the pax interchange d173 1 a173 1
The libarchive d179 1 a179 1
Only regular files, d186 1 a186 1
Path names in the archive are d191 1 a191 1
Symbolic links and hard links d197 1 a197 1
User and group information are d203 1 a203 1
Extended attributes, file d209 1 a209 1
Archive entries are limited to d212 1 a212 1
Generally, users should prefer d216 1 a216 1
The libarchive d223 1 a223 1
The POSIX standards require d236 1 a236 1
Libarchive recognizes ACL and d239 1 a239 1
The first tar d246 11 a256 13
Cpio Formats
The libarchive library can read and write a number of common cpio variants. A cpio archive stores each entry as a fixed-size header followed by a variable-length filename and variable-length data. Unlike the tar format, the cpio format does only minimal padding of the header or file data. There are several cpio variants, which differ primarily in how they store the initial header: some store the values as octal or hexadecimal numbers in ASCII, others as binary values of varying byte order and length.
d260 1 a260 1The libarchive d274 1 a274 1
This is the d287 1 a287 1
The libarchive library can read d295 1 a295 1
Cpio first d313 6 a318 8
Shar Formats
A “shell archive” is a shell script that, when executed on a POSIX-compliant system, will recreate a collection of file system objects. The libarchive library can write two different kinds of shar archives:
d322 1 a322 1The traditional d324 5 a328 6 echo(1), mkdir(1), and sed(1). It is suitable for portably archiving small collections of plain text files. However, it is not generally well-suited for large archives (many implementations of sh(1) have limits on the size of a script) nor should it be used with non-text files.
d332 23 a354 24This format is similar to shar but encodes files using uuencode(1) so that the result will be a plain text file regardless of the file contents. It also includes additional shell commands that attempt to reproduce as many file attributes as possible, including owner, mode, and flags. The additional commands used to restore file attributes make shardump archives less portable than plain shar archives.
ISO9660 format
Libarchive can read and extract from files containing ISO9660-compliant CDROM images. In many cases, this can remove the need to burn a physical CDROM just in order to read the files contained in an ISO9660 image. It also avoids security and complexity issues that come with virtual mounts and loopback devices. Libarchive supports the most common Rockridge extensions and has partial support for Joliet extensions. If both extensions are present, the Joliet extensions will be used and the Rockridge extensions will be ignored. In particular, this can create problems with hardlinks and symlinks, which are supported by Rockridge but not by Joliet.
d356 1 a356 1Libarchive reads d368 1 a368 1
Libarchive can d378 13 a390 2
Zip format
d392 1 a392 13Libarchive can read and write zip format archives that have uncompressed entries and entries compressed with the “deflate” algorithm. Other zip compression algorithms are not supported. It can extract jar archives, archives that use Zip64 extensions and self-extracting zip archives. Libarchive can use either of two different strategies for reading Zip archives: a streaming strategy which is fast and can handle extremely large archives, and a seeking strategy which can correctly process self-extracting Zip archives and archives with deleted members or other in-place modifications.
The streaming d405 1 a405 1
In particular, d416 11 a426 13
Archive (library) file format
The Unix archive format (commonly created by the ar(1) archiver) is a general-purpose format which is used almost exclusively for object files to be read by the link editor ld(1). The ar format has never been standardised. There are two common variants: the GNU format derived from SVR4, and the BSD format, which first appeared in 4.4BSD. The two differ primarily in their handling of filenames longer than 15 characters: the GNU/SVR4 variant writes a filename table at the beginning of the archive; the BSD format stores each d438 11 a448 13
mtree
Libarchive can read and write files in mtree(5) format. This format is not a true archive format, but rather a textual description of a file hierarchy in which each line specifies the name of a file and provides specific metadata about that file. Libarchive can read all of the keywords supported by both the NetBSD and FreeBSD versions of mtree(8), although many of the keywords cannot currently be stored in an archive_entry object. When writing, libarchive supports use of the archive_write_set_options(3) interface to specify d455 1 a455 1
When reading an d465 27 d493 1 a493 6
7-Zip
Libarchive can read and write 7-Zip format archives. TODO: Need more information
CAB
d495 3 a497 30Libarchive can read Microsoft Cabinet ( “CAB”) format archives. TODO: Need more information.
LHA
TODO: Information about libarchive’s LHA support
RAR
Libarchive has limited support for reading RAR format archives. Currently, libarchive can read RARv3 format archives which have been either created uncompressed, or compressed using any of the compression methods supported by the RARv3 format. Libarchive can also read self-extracting RAR archives.
Warc
Libarchive can read and write “web archives”. TODO: Need more information
XAR
Libarchive can read and write the XAR format used by many Apple tools. TODO: Need more information
SEE ALSO
d499 2 a500 5ar(1), cpio(1), mkisofs(1), shar(1), tar(1), zip(1), zlib(3), cpio(5), mtree(5), tar(5) Debian December 27, 2016 LIBARCHIVE-FORMATS(5)
@ 1.1.1.15 log @libarchive: imported version 3.7.9 @ text @d2 1 a2 1 @ 1.1.1.16 log @libarchive: import version 3.8.0 Libarchive 3.8.0 is a feature and bugfix release. New features: bsdtar: support --mtime and --clamp-mtime lib: mbedtls 3.x compatibility 7-zip reader: improve self-extracting archive detection xar: xmllite support for the XAR reader and writer zip writer: added XZ, LZMA, ZSTD and BZIP2 support zip writer: added LZMA + RISCV BCJ filter Notable security fixes: rar: do not skip past EOF while reading rar: fix double free with over 4 billion nodes rar: fix heap-buffer-overflow warc: prevent signed integer overflow tar: fix overflow in build_ustar_entry Notable bugfixes: bsdtar: don't hardlink negative inode files together gz: allow setting the original filename for gzip compressed files lib: improve lseek handling lib: support @@-prefixed Unix epoch timestamps as date strings rar: support large headers on 32 bit systems tar reader: Improve LFS support on 32 bit systems @ text @d2 1 a2 1 d391 3 a393 12 entries compressed with the “deflate” , “LZMA” , “XZ” , “BZIP2” and “ZSTD” algorithms. Libarchive can also read, but not write, zip format archives that have entries compressed with the “PPMd” algorithm. Other zip compression algorithms are not supported. The extensions supported by libarchive are Zip64, Libarchive’s extensions to better support streaming, PKZIP’s traditional ZIP encryption, Info-ZIP’s Unix extra fields, extra time, and Unicode path, as well as WinZIP’s AES encryption. It can extract jar archives, __MACOSX resource forks extension for OS X, and @ 1.1.1.17 log @libarchive: import version 3.8.1 @ text @d2 1 a2 1 @ 1.1.1.18 log @libarchive: imported version 3.8.2 @ text @d2 1 a2 1 @ 1.1.1.19 log @libarchive: import version 3.8.3 @ text @d2 1 a2 1 @ 1.1.1.20 log @libarchive: import 3.8.4 @ text @d2 1 a2 1 @ 1.1.1.21 log @libarchive: import version 3.8.5 @ text @d2 1 a2 1 @ 1.1.1.22 log @libarchive: imported version 3.8.6 @ text @d2 1 a2 1 @ 1.1.1.23 log @libarchive: imported version 3.8.7 @ text @d2 1 a2 1 @