Ubuntu 13.0 Highly Compressed Rar File 4 53
ZIP is an archive file format that supports lossless data compression. A ZIP file may contain one or more files or directories that may have been compressed. The ZIP file format permits a number of compression algorithms, though DEFLATE is the most common. This format was originally created in 1989 and was first implemented in PKWARE, Inc.'s PKZIP utility,[2] as a replacement for the previous ARC compression format by Thom Henderson. The ZIP format was then quickly supported by many software utilities other than PKZIP. Microsoft has included built-in ZIP support (under the name "compressed folders") in versions of Microsoft Windows since 1998 via the "Plus! 98" addon for Windows 98. Native support was added as of the year 2000 in Windows ME. Apple has included built-in ZIP support in Mac OS X 10.3 (via BOMArchiveHelper, now Archive Utility) and later. Most free operating systems have built in support for ZIP in similar manners to Windows and Mac OS X.
ubuntu 13.0 highly compressed rar file 4 53
.ZIP files are archives that store multiple files. ZIP allows contained files to be compressed using many different methods, as well as simply storing a file without compressing it. Each file is stored separately, allowing different files in the same archive to be compressed using different methods. Because the files in a ZIP archive are compressed individually, it is possible to extract them, or add new ones, without applying compression or decompression to the entire archive. This contrasts with the format of compressed tar files, for which such random-access processing is not easily possible.
Each entry stored in a ZIP archive is introduced by a local file header with information about the file such as the comment, file size and file name, followed by optional "extra" data fields, and then the possibly compressed, possibly encrypted file data. The "Extra" data fields are the key to the extensibility of the ZIP format. "Extra" fields are exploited to support the ZIP64 format, WinZip-compatible AES encryption, file attributes, and higher-resolution NTFS or Unix file timestamps. Other extensions are possible via the "Extra" field. ZIP tools are required by the specification to ignore Extra fields they do not recognize.
Tools that correctly read ZIP archives must scan for the end of central directory record signature, and then, as appropriate, the other, indicated, central directory records. They must not scan for entries from the top of the ZIP file, because (as previously mentioned in this section) only the central directory specifies where a file chunk starts and that it has not been deleted. Scanning could lead to false positives, as the format does not forbid other data to be between chunks, nor file data streams from containing such signatures. However, tools that attempt to recover data from damaged ZIP archives will most likely scan the archive for local file header signatures; this is made more difficult by the fact that the compressed size of a file chunk may be stored after the file chunk, making sequential processing difficult.
The extra field contains a variety of optional data such as OS-specific attributes. It is divided into records, each with at minimum a 16-bit signature and a 16-bit length. A ZIP64 local file extra field record, for example, has the signature 0x0001 and a length of 16 bytes (or more) so that two 64-bit values (the uncompressed and compressed sizes) may follow. Another common local file extension is 0x5455 (or "UT") which contains 32-bit UTC UNIX timestamps.
If the bit at offset 3 (0x08) of the general-purpose flags field is set, then the CRC-32 and file sizes are not known when the header is written. If the archive is in Zip64 format, the compressed and uncompressed size fields are 8 bytes long instead of 4 bytes long (see section 4.3.9.2[33]). The equivalent fields in the local header (or in the Zip64 extended information extra field in the case of archives in Zip64 format) are filled with zero, and the CRC-32 and size are appended in a 12-byte structure (optionally preceded by a 4-byte signature) immediately after the compressed data:
The original .ZIP format had a 4 GB (232 bytes) limit on various things (uncompressed size of a file, compressed size of a file, and total size of the archive), as well as a limit of 65,535 (216-1) entries in a ZIP archive. In version 4.5 of the specification (which is not the same as v4.5 of any particular tool), PKWARE introduced the "ZIP64" format extensions to get around these limitations, increasing the limits to 16 EB (264 bytes). In essence, it uses a "normal" central directory entry for a file, followed by an optional "zip64" directory entry, which has the larger fields.[39]
A Seek-Optimized ZIP file (SOZip) profile [48] has been proposed for the ZIP format. Such file contains one or several Deflate-compressed files that are organized and annotated such that a SOZip-aware reader can perform very fast random access (seek) within a compressed file. SOZip makes it possible to access large compressed files directly from a .zip file without prior decompression. It combines the use of ZLib block flushs issued at regular interval with a hidden index file mapping offsets of the uncompressed file to offsets in the compressed stream. ZIP readers that are not aware of that extension can read a SOZip-enabled file normally and ignore the extended features that support efficient seek capability.
zpaq is a free and open sourceincremental, journaling command-line archiver for Windows, Linux and Mac OS/X.Incremental means that when you back up your hard drive, forexample: zpaq add e:\backup.zpaq c:\*then only those files whose last-modified date or size has changed sincethe previousbackup are added. For 100 GB of files, this typically takes about a minute, vs.an hour to create the first version.Journaling means that the archive is append-only. When you addfiles or directories to the archive, both the old and new versions are saved.You can recover old versions by specifying the date or version number,for example: zpaq extract e:\backup.zpaq c:\Users\Bob -to tmp -until 2013-10-30will extract all the files and directories in c:\Users\Bob as of the lastbackup on or before Oct. 30, 2013 and put them in a directory named tmp.zpaq is faster and compresses better than most other popular archiversand backup programs,especially for realistic backups that have a lot of duplicate filesand a lot of already compressed files.Archive size vs. time tocompress and extract 10 GB (79,431 files) to an external USB harddrive at default and maximumsettings on a Dell Latitude E6510 laptop (Core i7 M620, 2+2hyperthreads, 2.66 GHz, 4 GB, Ubuntu Linux, Wine 1.6).Data from10 GB Benchmark (system 4).Feature comparison zpaq pcompress exdupe freearc obnam rar 7zip zipWindows W W W W W WLinux L L L L L L L LUpdate U U U U U U UIncremental I I I I IRollback R RDedupe D D D DEncryption E E E E E EGUI G G G GFree F F F F F F FOpen source O O O O O O OSpecification S S Downloadzpaq.exe for Windows.The latest version is zpaq v7.15,released Aug. 17, 2016. The downloadcontain source code (zpaq.cpp, libzpaq.cpp, libzpaq.h), Windows executables(32 or 64 bit, XP or later), documentation (zpaq.pod),and a Makefile for compiling in Linux, BSD, or Mac OS/X.You may need unzip.exe to unzip from theWindows command line.zpaq man page (HTML, latest version).The ZPAQ archive format is described by aspecification and reference decoder.A test caseexercising all of the specification features should decompress to theCalgary corpus.The compression algorithm is describedhere.The source code includes the libzpaq API providing compressionand decompression services for applications in C++.Developers may be interested in thezpaqd development tool and sample configuration filesfound on the utilities page.zpaq is written by Matt Mahoney and released to the public domain.It includes code from libdivsufsort 2.0 (C) Yuta Mori, 2003-2008, MIT license,public domain code for AES from libtomcrypt by Tom St Denis andpublic domain code for salsa20 by D. J. Bernstein.FeaturesA zpaq archive can contain at most 4 billion files and at most250 terabytes of data after deduplication and before compression.zpaq is for user-level backups. Do not use it to back up the operatingsystem or any software that requires a password to install.zpaq saves regular files and directories, last-modified dates (to thenearest second), and (optionally) Windows attributes or Linuxpermissions. It does not follow or save symbolic links or junctions.It unknowingly follows hard links. It does not save owner or group IDs,ACLs, extended attributes,the registry, or special file types like devices, sockets, or named pipes.Open standard specificationThe zpaq archive format is described by a precisespecification and reference decoder (above).The format is not encumbered by any patents or pending patentsin any country as far as I know. I have purposely published allpast versions (below) to establish prior art so that no patents can be filed.Backward and forward compatibilityAll versions of zpaq can read archives produced by older versionsback to version 1.00 (March 2009). To some extent, older versions canread archives produced by newer versions (forward compatibility) providedthey don't use any unsupported features. These are as follows:v1.00 (Mar. 2009). Level 1 format. Streaming archives with at leastone context model. Does not support deduplication or rollback.v5.00 (Aug. 2012). Level 2 format. Adds support for compressionwith pre/post processing with no context modeling (e.g. uncompressed or LZ77).v6.00 (Sept. 2012). Journaling format (dedupe and rollback).v6.44 (Jan. 2014). Encrypted archives.v6.47 (Jan. 2014). Multi-part archives. Older versions can read them if concatenated.Many intermediate versions include compression improvements. This doesnot break forward compatibility because the decompression codeis stored in the archive. The code is written in a sandboxed,virtual machine language called ZPAQL. On x86-32 and x86-64processors, the ZPAQL code is translated to machine code and executed,so it is as fast as compression algorithms written in compiledlanguages like C or C++. On other hardware, the ZPAQL code is interpreted,which takes about twice as long.For example, the following will create a streaming archive using BWTcompression that can be extracted by all versions back to v1.00, even thoughmost of these versions could not compress using BWT. zpaq add archive.zpaq files -method s4.3ci1RollbackAn archive is updated only by appending changes to it. You canroll back the archive to an earlier state by using the -until optionto specify the date and time or version number where to stop reading.When updating, -until will truncate the archive at thatpoint before appending. So if you backed up some files you didn't meanto, then you can truncate the last update and repeat: zpaq add backup c:\ -not c:\tmp -until -1Transacted updatesUpdates are committed by first appending a temporary header and thenupdating it when all of the compressed data and index changes are appended.If you interrupt zpaq (by typing Ctrl-C), then the partially appendeddata will be ignored and overwritten on the next update.DeduplicationWhen adding files,zpaq uses a rolling hash function to split files into fragmentswith an average size of 64 KB along content-dependent boundaries.Then it computes the SHA-1 hash of the fragment and compares it withsaved hashes from the current and previous versions.If it finds a match then the fragment is not stored.Deduplication requires 1 MB of memory per GB of deduplicatedbut uncompressed archive data to update, and 0.5 MB per GB tolist or extract.Incremental update and restoreFiles are added only if the date has changed since the last update.You can use the -force option to override, but in this case the filewill be deduplicated and not saved unless the contents have really changed.This is slower than comparing dates but faster than compressing it again.Extraction will not clobber existing files unless you give the-force option to allow overwrite. In this case, the file to be overwrittenis compared with the stored hashes and not decompressed unless the sizeor contents is different.Remote archive supportzpaq updates an archive by appending changes to it. To supportremote backups without having to move huge files,zpaq can put the appended changes into a separate,numbered file that you would copy or move to remote storage.You can concatenate the parts to form a complete archive, or simplyread them all at once by specifying a pattern in the archive namelike "part???.zpaq".zpaq will then search for part001.zpaq, part002.zpaq, etc. and regardthe concatenated sequence as a single archive.To make incremental backups with a local copy: zpaq add "arc???" files (copy arc001.zpaq) zpaq add "arc???" files (copy arc002.zpaq) zpaq list "arc???" (show contents) zpaq extract "arc???" (restore)To back up without keeping a local copy of the archive, you keepa small local index (arc000.zpaq) as a copy of the remote archive minusthe compressed file contents. zpaq maintains consistency between theindex and archive. zpaq add "arc???" files -index arc000.zpaq (move arc001.zpaq) zpaq add "arc???" files -index arc000.zpaq (move arc002.zpaq) zpaq list arc000 (show arc???.zpaq contents)EncryptionArchives can be encrypted using AES-256 in CTR mode.A password must be given every time an encrypted archive is used.Keys are strengthened with Scrypt(N=16384, r=8, p=1)(requiring 208M operations and 16 MB memory)to slow down brute force search for weakkeys. Encrypted archives are prefixed with a 32 byte random salt, whichalso provides an 8 byte IV for the first half of the 16 byte AES counter.If a remote archive has a local index, then both are encrypted with thesame key but different salts to generate independent keystreams.Encryption provides privacy but not authentication against tampering.All of the encryption code (AES, Scrypt, SHA-1, SHA-256)is public domain and tested against published test vectors.The AES code is derived from libtomcrypt 1.17.Multithreaded compressionzpaq has 5 compression levels. The default, -method 1, is the fastest.It is best for backups where you compress often and extract rarely.-method 2 compresses slower but decompresses as fast as -method 1.It is best for distributing files where you compress once and extractoften. Methods 3, 4, and 5 are slower with better compression.Fragments not removed by deduplication are packed into blocksfor compression. Files are sorted by filename extension and then bydecreasing size in order to group similar files together.The block size is 16 MB for method 1 and 64 MB for higher methods.You can change the block size to trade compression for memory usage.Blocks are compressed or decompressed in parallel in separate threads.zpaq automatically detects the number of processor cores and uses all of themin the 64 bit version or at most 2 in the 32 bit version (which is limitedto 2 GB memory).You can use the -threads option to change the number of threads.Resident memory per thread required to compress or decompress isapproximately as follows. Virtual memory usage may be higher. Method Compress Decompress Algorithm ------ -------- ---------- --------- 1 128 MB 32 MB LZ77 2 450 MB 128 MB LZ77 3 450 MB 400 MB LZ77+CM or BWT 4 550 MB 550 MB LZ77+CM, BWT or CM 5 850 MB 850 MB CMMethod 1 uses LZ77, compressing by replacing duplicate strings withpointers to previous occurrences. Method 2 is the same but spendsmore time looking for better matches (using a suffix array instead ofa hash table). Method 3 uses either BWT (context sorting) orLZ77 for long matches and an order 1 context model and arithmeticcoding for literals depending on the file type. Method 4 either uses LZ77, BWTor a high order context model. Method 5 uses a complex, high order contextmixing model with over 20 bit prediction components.All methods except 5 test whether the data appears to be compressibleor already compressed (random). Uncompressible data is simply stored.An E8E9 filter is applied if x86 data (normally found in .exe and .dllfiles) is detected. The filter replaces x86 CALL and JMP relativeaddresses with absolute addresses to make the data more compressible.Data analysiszpaq has list options to make it easier to examine the contentsof archives containing millions of files. For example, the followingcompares external dir1 to internal dir2 and lists only differences.Files are compared quickly by size and last modified date, or thoroughlyby reading the file, computing its SHA-1 hashes and comparing withthe hashes stored in the archive. zpaq list backup dir1 -to dir2 -not = (compare dates) zpaq list backup dir1 -to dir2 -not = -force (compare contents)Other useful list options: -only *.exe List only files ending with .exe -not *.exe Don't list files matching a pattern. -summary 20 List the 20 largest files and identify duplicates. -all Show all file versions. -until 20 List contents as of the 20'th updateError detection and recoveryzpaq archives are designed to minimize data loss if damaged. Anarchive is divided into blocks that can be decompressed independently.Each block begins with a 13 byte tag that can be found by scanningif the previous block is damaged. Each block ends with the SHA-1 hashof the uncompressed data, which is verified to detect errors. Blockswith hash mismatches or other errors are ignored with a warningwithout killing zpaq.Each update contains 4 types of blocks.C - Update header: date, size of compressed data.D - Compressed data fragments, list of fragment sizes.H - List of fragment hashes and sizes, one per D block.I - Index updates: list of files updated or deleted. Each update includes the date, attributes, and list of fragments.C blocks are used to skip over D blocks to read the index quickly.They are not needed to extract.If a D or H block is lost then so are any files that point to it.If an I block is lost, then so areany files in it. I blocks are small (16 KB) to minimize damage.When extracting files, the D block is decompressed up to the lastused fragment and those fragments are hashed and compared to thestored hashes in the H block.The zpaq -test -all extract option will decompress internallyand verify all of the fragment hashes without writing the output files.Public Domain APIThe source download includes libzpaq,a public domain application programming interface (API) in C++ thatprovides streaming compression and decompression servicesto and from files, strings, or arrays using built-in and customcompression algorithms. To use the code, you include libzpaq.hin your program and link to libzpaq.cpp.The API documentation is in libzpaq.h. The precise semantics isdescribed in the ZPAQ specification.In the simplest case,the application provides an error handling function and derivedimplementations of two abstract classes, Reader and Writer,specifying the input and output byte streams. For example, to compressfrom stdin to stdout (assuming binary I/O as in Linux): #include "libzpaq.h" #include #include void libzpaq::error(const char* msg) // print message and exit fprintf(stderr, "Oops: %s\n", msg); exit(1); class In: public libzpaq::Reader public: int get() return getchar(); // returns byte 0..255 or -1 at EOF in; class Out: public libzpaq::Writer public: void put(int c) putchar(c); // writes 1 byte 0..255 out; int main() libzpaq::compress(&in, &out, "1"); // -method 1 To decompress: libzpaq::decompress(&in, &out);There are also functions for reading and writing block and segmentheaders and for passing specialized methods or ZPAQL code to the compressor,as documented in libzpaq.h.The ZPAQ utilities pagecontains sample compression algorithms written in ZPAQLand a tool zpaqd for running, testing, and debugging ZP