diff --git a/bin/precomp_w_preflate_msvc17_aka_vc140_x64.exe b/bin/precomp_w_preflate_msvc17_aka_vc140_x64.exe index 9a7c67c..0a59029 100644 Binary files a/bin/precomp_w_preflate_msvc17_aka_vc140_x64.exe and b/bin/precomp_w_preflate_msvc17_aka_vc140_x64.exe differ diff --git a/bin/precomp_w_preflate_msvc17_aka_vc140_x86.exe b/bin/precomp_w_preflate_msvc17_aka_vc140_x86.exe index b38449e..4997166 100644 Binary files a/bin/precomp_w_preflate_msvc17_aka_vc140_x86.exe and b/bin/precomp_w_preflate_msvc17_aka_vc140_x86.exe differ diff --git a/contrib/preflate/README.md b/contrib/preflate/README.md index 3ffcca9..8a14570 100644 --- a/contrib/preflate/README.md +++ b/contrib/preflate/README.md @@ -1,4 +1,4 @@ -preflate v0.1.2 +preflate v0.3.1 =============== Library to split deflate streams into uncompressed data and reconstruction information, or reconstruct the original deflate stream from those two. @@ -15,19 +15,31 @@ Reconstructing the original deflate stream becomes important if the position or of the reconstructed deflate streams must not differ, e.g. if those streams are embedded into executables, or unsupported archive files where the indices cannot be adapted correctly. -There are currently at least two tools available which try to solve this problem: +There are currently already some other tools available which try to solve this problem: - "precomp" is a tool which can do the bit-correct reconstruction very efficiently, - but only for deflate streams that were created by the zlib library. (It only needs - to store the three relevant zlib parameters to allow reconstruction.) + but only for deflate streams that were created by the ZLIB library. (It only needs + to store the three relevant ZLIB parameters to allow reconstruction.) It bails out for anything created by 7zip, kzip, or similar tools. Of course, "precomp" also handles JPG, PNG, ZIP, GIF, PDF, MP3, etc, which makes a very nice tool, and it is open source. - "reflate" can reconstruct any deflate stream (also 7zip, kzip, etc), but it is only - efficient for streams that were created by zlib, compression level 9. - All lower compression levels of zlib require increasing reconstruction info, the further + close to perfect for streams that were created by ZLIB, compression level 9. + All lower compression levels of ZLIB require increasing reconstruction info, the further from level 9, the bigger the required reconstruction data. "reflate" only handles deflate streams, and is not open source. As far as I know, it is also part of PowerArchiver. +- "grittibanzli" is also open source, and can reconstruct any deflate stream. + (It was published after the first version of "preflate".) + It outputs the reconstruction data (the diff against the prediction) in + an uncompressed, byte based format, and relies on an external compressor to squeeze it + down. This work quite well, and in my tests, the compressed reconstruction data is + usually only 20 to 30% bigger than "preflate"s, and smaller than "reflate"s. + The ZLIB level detection is rather basic (worse then "preflate"s, and much worse than + "precomp"s), and for streams which were compressed with a lower level of ZLIB, it + will create reconstruction data that are not just a few bytes. + In my tests, compression time was the same for "grittibanzli" and "preflate", while + for decompression it was faster. However, the higher memory requirements will put + a strain on some systems. What about "difflate"? @@ -40,38 +52,46 @@ development and not yet feature complete. Let's wait and see. So, what is the point of "preflate"? ------------------------------------ The goal of "preflate" is to get the best of both worlds: -- for deflate streams created by zlib at any compression level, we want to +- for deflate streams created by ZLIB at any compression level, we want to be able to reconstruct it with only a few bytes of reconstruction information (like "precomp") - for all other deflate streams, we want to be able to reconstruct them with reconstruction information that is not much larger than "reflate" -Right now, it has been tested on 11159 valid deflate streams, extracted with -"rawdet" from archives etc., and preflate was capable of inflating and reconstructing -all of them. +Right now, it has been tested on some ten thousand valid deflate streams, some extracted with +"rawdet" from archives etc., and "preflate" was capable of inflating and reconstructing +all of them. There's also an experimental fork of "precomp" which incorporates it, +where it has to cope with lots of invalid deflate streams, or random data which just +looks like a deflate streams. It was tested on several GiB of data with tens of +thousands of valid and invalid deflate streams, and it seems to work quite reliable, +albeit not perfect. -There are still known (and also likely unknown) corner cases of valid deflate stream in -which preflate will fail. +There might be some unknown corner cases of valid deflate stream in which preflate +will fail. And there are probably some cases of invalid deflate streams which will +make it crash. So, is "preflate" already better than "precomp" and "reflate"? -------------------------------------------------------------- -No. It isn't. +No. It isn't. However, it is a reasonable alternative. -Test coverage is still quite low, and testing it is currently quite cumbersome -(because it only works on raw deflate streams which need to be extracted first.) +First, "preflate" only works on raw deflate streams. So, it is not intended as +a tool, but as a library to be used by some other tool. (Did I mention the "precomp" +fork?) The current ZLIB level detection inside "precomp" is much better than +"preflate"s (and it needs to be), so there are a considerable amount of ZLIB deflate +streams, probably 20-30%, for which "preflate" generates reconstruction data that +is bigger than 3 bytes (usually only a few tens of bytes though). +Or in other words: there are files in which Vanilla "precomp" BEATS "preflate" +in SIZE. -It's very slow. (50-500% slower than "precomp" or "reflate"). Especially for files -containing long runs of the same byte (e.g. \0 or spaces), it gets very, very slow. +It's quite slow. Since 0.2.1, there is some optimization for long runs of the same byte (e.g. +\0 or spaces), and some files profit from that tremendously (which were ten times +slower than now), but for most files the gain is only a few percent. +Both "precomp" and "reflate" BEAT "preflate" in SPEED. (Expected to be around +50-500%). -It will not handle all valid deflate streams. (E.g., preflate will fail if the reference -length 258 is encoded as 227+31.) I don't know if this is really a problem. All good -encoders would never encode a length of 227+31 anyway. - -The detection of the zlib compression parameters is not always on spot, which leads to -the creation of larger diffs than necessary. - -Right now, it is a proof of concept, that we can do better than both "precomp" and "reflate". -It just isn't stable and fast enough yet to be of practical use. +However, "preflate" eats anything and can be successfully applied to files in which +"precomp" will fail completely. It generates smaller reconstruction data than +"reflate" (in my tests). And it is open source. How do I build it? @@ -79,15 +99,19 @@ How do I build it? There is a make file, but it has only been tested so far with MinGW gmake. The produced executable is larger than 1MiB, while the MSVC compiler generated executables were around 100KiB. The reason for that is unclear at the moment. +There is also a CMake script which hopefully works for non Windows platforms. Credits ------- - "precomp" by Christian Schneider + (https://github.com/schnaader/precomp-cpp) - "reflate" and "rawdet" by Eugene Shelwien - "zlib" by Mark Adler et al. - "7zip" by Igor Pavlov - "kzip" by Ken Silverman +- "grittibanzli" by Google Zurich (inofficial project) + (https://github.com/google/grittibanzli) All of the software above is just AWESOME! @@ -102,28 +126,25 @@ Contains information about a lot of interesting compression tools of which I pro would never have known without this site. Also, Stephan helped getting rid of several bugs in preflate. Thank you. +- packARI by Matthias Stirner -Notes ------ -Currently, "preflate" uses code from two libraries: -- packARI by Matthias Stirner, which is licensed under LGPL3 +packARI provides context-aware, adaptive order(n) encoding. "preflate"s +arithmetic coder is based on packARI's algorithms, but only uses static +models and contains some speed optimizations which are totally pointless because +"preflate" spends all its time in the match finder. - (directory packARI) - It is used for the arithmetic coding of the reconstruction information. -- zlib 1.2.11 by Mark Adler et al., under the zlib license. - - (directory zlib 1.2.11.dec. Does NOT contain the full zlib library!) - - It is used for the decoding of deflate streams, and some callbacks were - added to get the decoded trees and tokens, which are then used to build - the reconstruction information. - -The usage of both libraries will be removed in the future. -There already is a new implementation of deflate decoding (which is still slower -than the zlib one with callbacks). -And packARI is much more powerful and flexible than what is actually needed -in "preflate" right now. +Changes +------- +- 0.1.0 - first public release +- 0.2.0 - freeze bitstream format, remove zlib and packARI dependencies +- 0.2.1 - add match finder for same character sequences +- 0.3.0 - split up large deflate streams into "meta-blocks" of a few megabyte each + (this should help keep memory usage in check) +- 0.3.1 - mitigation against zlib level estimation failure due to + "meta block splitting". For one 350KiB file compressed at zlib level 1, + reconstruction size goes from 35KiB to 1.5KiB. Without meta-blocks it + would be 3 bytes... License diff --git a/contrib/preflate/preflate_block_reencoder.cpp b/contrib/preflate/preflate_block_reencoder.cpp index 7860ac2..cdf68a8 100644 --- a/contrib/preflate/preflate_block_reencoder.cpp +++ b/contrib/preflate/preflate_block_reencoder.cpp @@ -20,10 +20,11 @@ PreflateBlockReencoder::PreflateBlockReencoder( BitOutputStream& bos, - const std::vector& uncompressedData) + const std::vector& uncompressedData, + const size_t uncompressedOffset) : _output(bos) , _uncompressedData(uncompressedData) - , _uncompressedDataPos(0) + , _uncompressedDataPos(uncompressedOffset) , _errorCode(OK) , _dynamicLitLenEncoder(nullptr, 0, false) , _dynamicDistEncoder(nullptr, 0, false) { diff --git a/contrib/preflate/preflate_block_reencoder.h b/contrib/preflate/preflate_block_reencoder.h index e8918be..09d0fa4 100644 --- a/contrib/preflate/preflate_block_reencoder.h +++ b/contrib/preflate/preflate_block_reencoder.h @@ -46,7 +46,9 @@ public: const unsigned short *litLenCode, *distCode, *treeCode; const unsigned char *litLenBits, *distBits, *treeBits;*/ - PreflateBlockReencoder(BitOutputStream& bos, const std::vector& uncompressedData); + PreflateBlockReencoder(BitOutputStream& bos, + const std::vector& uncompressedData, + const size_t uncompressedOffset); bool writeBlock(const PreflateTokenBlock&, const bool last); void flush(); diff --git a/contrib/preflate/preflate_checker.cpp b/contrib/preflate/preflate_checker.cpp index 1e4d800..51f0680 100644 --- a/contrib/preflate/preflate_checker.cpp +++ b/contrib/preflate/preflate_checker.cpp @@ -58,7 +58,7 @@ bool preflate_checker(const std::vector& deflate_raw) { printf("Unpacked data has size %d\n", (int)unpacked_output.size()); // Encode - PreflateParameters paramsE = estimatePreflateParameters(unpacked_output, blocks); + PreflateParameters paramsE = estimatePreflateParameters(unpacked_output, 0, blocks); printf("prediction parameters: w %d, c %d, m %d, zlib %d, farL3M %d, very far M %d, M2S %d, log2CD %d\n", paramsE.windowBits, paramsE.compLevel, paramsE.memLevel, paramsE.zlibCompatible, paramsE.farLen3MatchesDetected, @@ -67,8 +67,8 @@ bool preflate_checker(const std::vector& deflate_raw) { PreflateStatisticsCounter counterE; memset(&counterE, 0, sizeof(counterE)); - PreflateTokenPredictor tokenPredictorE(paramsE, unpacked_output); - PreflateTreePredictor treePredictorE(unpacked_output); + PreflateTokenPredictor tokenPredictorE(paramsE, unpacked_output, 0); + PreflateTreePredictor treePredictorE(unpacked_output, 0); for (unsigned i = 0, n = blocks.size(); i < n; ++i) { tokenPredictorE.analyzeBlock(i, blocks[i]); if (tokenPredictorE.predictionFailure) { @@ -124,7 +124,7 @@ bool preflate_checker(const std::vector& deflate_raw) { printf("Prediction diff has size %d\n", (int)preflate_diff.size()); // Decode - PreflateMetaDecoder codecD(preflate_diff, unpacked_output); + PreflateMetaDecoder codecD(preflate_diff, unpacked_output.size()); PreflatePredictionDecoder pcodecD; PreflateParameters paramsD; if (codecD.error() || codecD.metaBlockCount() != 1) { @@ -163,13 +163,13 @@ bool preflate_checker(const std::vector& deflate_raw) { return false; } - PreflateTokenPredictor tokenPredictorD(paramsD, unpacked_output); - PreflateTreePredictor treePredictorD(unpacked_output); + PreflateTokenPredictor tokenPredictorD(paramsD, unpacked_output, 0); + PreflateTreePredictor treePredictorD(unpacked_output, 0); MemStream mem; BitOutputStream bos(mem); - PreflateBlockReencoder deflater(bos, unpacked_output); + PreflateBlockReencoder deflater(bos, unpacked_output, 0); unsigned blockno = 0; bool eof = true; do { diff --git a/contrib/preflate/preflate_complevel_estimator.cpp b/contrib/preflate/preflate_complevel_estimator.cpp index 8ff6d42..563d005 100644 --- a/contrib/preflate/preflate_complevel_estimator.cpp +++ b/contrib/preflate/preflate_complevel_estimator.cpp @@ -21,6 +21,7 @@ PreflateCompLevelEstimatorState::PreflateCompLevelEstimatorState( const int wbits, const int mbits, const std::vector& unpacked_output_, + const size_t off0_, const std::vector& blocks_) : slowHash(unpacked_output_, mbits) , fastL1Hash(unpacked_output_, mbits) @@ -28,9 +29,11 @@ PreflateCompLevelEstimatorState::PreflateCompLevelEstimatorState( , fastL3Hash(unpacked_output_, mbits) , blocks(blocks_) , wsize(1 << wbits) + , off0(off0_) { memset(&info, 0, sizeof(info)); info.possibleCompressionLevels = (1 << 10) - (1 << 1); + updateHash(off0); } void PreflateCompLevelEstimatorState::updateHash(const unsigned len) { @@ -98,6 +101,9 @@ bool PreflateCompLevelEstimatorState::checkMatchSingleFastHash( return true; } void PreflateCompLevelEstimatorState::checkMatch(const PreflateToken& token) { + if (slowHash.input().pos() < token.dist + off0) { + return; + } unsigned hashHead = slowHash.curHash(); if (info.possibleCompressionLevels & (1 << 1)) { if (!checkMatchSingleFastHash(token, fastL1Hash, fastPreflateParserSettings[0], hashHead)) { @@ -204,9 +210,10 @@ PreflateCompLevelInfo estimatePreflateCompLevel( const int wbits, const int mbits, const std::vector& unpacked_output, + const size_t off0, const std::vector& blocks, const bool early_out) { - PreflateCompLevelEstimatorState state(wbits, mbits, unpacked_output, blocks); + PreflateCompLevelEstimatorState state(wbits, mbits, unpacked_output, off0, blocks); state.checkDump(early_out); state.recommend(); return state.info; diff --git a/contrib/preflate/preflate_complevel_estimator.h b/contrib/preflate/preflate_complevel_estimator.h index 40ac138..fc6fb87 100644 --- a/contrib/preflate/preflate_complevel_estimator.h +++ b/contrib/preflate/preflate_complevel_estimator.h @@ -42,9 +42,11 @@ struct PreflateCompLevelEstimatorState { const std::vector& blocks; PreflateCompLevelInfo info; uint16_t wsize; + size_t off0; PreflateCompLevelEstimatorState(const int wbits, const int mbits, const std::vector& unpacked_output, + const size_t off0, const std::vector& blocks); void updateHash(const unsigned len); void updateOrSkipHash(const unsigned len); @@ -67,6 +69,7 @@ PreflateCompLevelInfo estimatePreflateCompLevel( const int wbits, const int mbits, const std::vector& unpacked_output, + const size_t off0, const std::vector& blocks, const bool early_out); diff --git a/contrib/preflate/preflate_decoder.cpp b/contrib/preflate/preflate_decoder.cpp index acbb700..320916c 100644 --- a/contrib/preflate/preflate_decoder.cpp +++ b/contrib/preflate/preflate_decoder.cpp @@ -24,18 +24,110 @@ #include "support/memstream.h" #include "support/outputcachestream.h" -bool preflate_decode(std::vector& unpacked_output, +class PreflateDecoderHandler : public PreflateDecoderTask::Handler { +public: + PreflateDecoderHandler(std::function progressCallback_) + : progressCallback(progressCallback_) {} + + bool finish(std::vector& reconstructionData) { + reconstructionData = encoder.finish(); + return !encoder.error(); + } + bool error() const { + return encoder.error(); + } + + virtual uint32_t setModel(const PreflateStatisticsCounter& counters, const PreflateParameters& parameters) { + return encoder.addModel(counters, parameters); + } + virtual bool beginEncoding(const uint32_t metaBlockId, PreflatePredictionEncoder& codec, const uint32_t modelId) { + return encoder.beginMetaBlockWithModel(codec, modelId); + } + virtual bool endEncoding(const uint32_t metaBlockId, PreflatePredictionEncoder& codec, const size_t uncompressedSize) { + return encoder.endMetaBlock(codec, uncompressedSize); + } + virtual void markProgress() { + progressCallback(); + } + +private: + PreflateMetaEncoder encoder; + std::function progressCallback; +}; + +PreflateDecoderTask::PreflateDecoderTask(PreflateDecoderTask::Handler& handler_, + const uint32_t metaBlockId_, + std::vector&& tokenData_, + std::vector&& uncompressedData_, + const size_t uncompressedOffset_, + const bool lastMetaBlock_, + const uint32_t paddingBits_) + : handler(handler_) + , metaBlockId(metaBlockId_) + , tokenData(tokenData_) + , uncompressedData(uncompressedData_) + , uncompressedOffset(uncompressedOffset_) + , lastMetaBlock(lastMetaBlock_) + , paddingBits(paddingBits_) { +} + +bool PreflateDecoderTask::execute() { + PreflateParameters params = estimatePreflateParameters(uncompressedData, uncompressedOffset, tokenData); + PreflateStatisticsCounter counter; + memset(&counter, 0, sizeof(counter)); + PreflateTokenPredictor tokenPredictor(params, uncompressedData, uncompressedOffset); + PreflateTreePredictor treePredictor(uncompressedData, uncompressedOffset); + for (unsigned i = 0, n = tokenData.size(); i < n; ++i) { + tokenPredictor.analyzeBlock(i, tokenData[i]); + treePredictor.analyzeBlock(i, tokenData[i]); + if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { + return false; + } + tokenPredictor.updateCounters(&counter, i); + treePredictor.updateCounters(&counter, i); + handler.markProgress(); + } + counter.block.incNonZeroPadding(paddingBits != 0); + PreflatePredictionEncoder pcodec; + unsigned modelId = handler.setModel(counter, params); + if (!handler.beginEncoding(metaBlockId, pcodec, modelId)) { + return false; + } + for (unsigned i = 0, n = tokenData.size(); i < n; ++i) { + tokenPredictor.encodeBlock(&pcodec, i); + treePredictor.encodeBlock(&pcodec, i); + if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { + return false; + } + if (lastMetaBlock) { + tokenPredictor.encodeEOF(&pcodec, i, i + 1 == tokenData.size()); + } + } + if (lastMetaBlock) { + pcodec.encodeNonZeroPadding(paddingBits != 0); + if (paddingBits != 0) { + unsigned bitsToSave = bitLength(paddingBits); + pcodec.encodeValue(bitsToSave, 3); + if (bitsToSave > 1) { + pcodec.encodeValue(paddingBits & ((1 << (bitsToSave - 1)) - 1), bitsToSave - 1); + } + } + } + return handler.endEncoding(metaBlockId, pcodec, uncompressedData.size() - uncompressedOffset); +} + +bool preflate_decode(OutputStream& unpacked_output, std::vector& preflate_diff, uint64_t& deflate_size, InputStream& deflate_raw, std::function block_callback, - const size_t min_deflate_size) { + const size_t min_deflate_size, + const size_t metaBlockSize) { deflate_size = 0; uint64_t deflate_bits = 0; size_t prevBitPos = 0; BitInputStream decInBits(deflate_raw); - MemStream decUnc; - OutputCacheStream decOutCache(decUnc); + OutputCacheStream decOutCache(unpacked_output); PreflateBlockDecoder bdec(decInBits, decOutCache); if (bdec.status() != PreflateBlockDecoder::OK) { return false; @@ -43,6 +135,14 @@ bool preflate_decode(std::vector& unpacked_output, bool last; unsigned i = 0; std::vector blocks; + std::vector blockSizes; + uint64_t sumBlockSizes = 0; + uint64_t lastEndPos = 0; + uint64_t uncompressedMetaStart = 0; + size_t MBSize = std::min(std::max(metaBlockSize, 1 << 18), (1 << 31) - 1); + size_t MBThreshold = (MBSize * 3) >> 1; + PreflateDecoderHandler encoder(block_callback); + size_t MBcount = 0; do { PreflateTokenBlock newBlock; @@ -50,74 +150,107 @@ bool preflate_decode(std::vector& unpacked_output, if (!ok) { return false; } - blocks.push_back(newBlock); - ++i; - if (decOutCache.cacheSize() >= 512 * 1024) { - decOutCache.flushUpTo(decOutCache.cacheEndPos() - (32 * 1024)); + + uint64_t blockSize = decOutCache.cacheEndPos() - lastEndPos; + lastEndPos = decOutCache.cacheEndPos(); + if (blockSize >= (1 << 31)) { + // No mega blocks + return false; } + + blocks.push_back(newBlock); + blockSizes.push_back(blockSize); + ++i; + block_callback(); + deflate_bits += decInBits.bitPos() - prevBitPos; prevBitPos = decInBits.bitPos(); - block_callback(); + + sumBlockSizes += blockSize; + if (last || sumBlockSizes >= MBThreshold) { + size_t blockCount, blockSizeSum; + if (last) { + blockCount = blockSizes.size(); + blockSizeSum = sumBlockSizes; + } else { + blockCount = 0; + blockSizeSum = 0; + for (const auto bs : blockSizes) { + blockSizeSum += bs; + ++blockCount; + if (blockSizeSum >= MBSize) { + break; + } + } + } + std::vector blocksForMeta; + for (size_t j = 0; j < blockCount; ++j) { + blocksForMeta.push_back(std::move(blocks[j])); + } + blocks.erase(blocks.begin(), blocks.begin() + blockCount); + blockSizes.erase(blockSizes.begin(), blockSizes.begin() + blockCount); + sumBlockSizes -= blockSizeSum; + + size_t uncompressedOffset = MBcount == 0 ? 0 : 1 << 15; + + std::vector uncompressedDataForMeta( + decOutCache.cacheData(uncompressedMetaStart - uncompressedOffset), + decOutCache.cacheData(uncompressedMetaStart - uncompressedOffset) + blockSizeSum + uncompressedOffset); + uncompressedMetaStart += blockSizeSum; + + size_t paddingBits = 0; + if (last) { + uint8_t remaining_bit_count = (8 - deflate_bits) & 7; + paddingBits = decInBits.get(remaining_bit_count); + + deflate_bits += decInBits.bitPos() - prevBitPos; + prevBitPos = decInBits.bitPos(); + } + + PreflateDecoderTask task(encoder, MBcount, + std::move(blocksForMeta), + std::move(uncompressedDataForMeta), + uncompressedOffset, + last, paddingBits); + if (!task.execute()) { + return false; + } + MBcount++; + + if (!last) { + decOutCache.flushUpTo(uncompressedMetaStart - (1 << 15)); + } + } } while (!last); decOutCache.flush(); - unpacked_output = decUnc.extractData(); deflate_size = (deflate_bits + 7) >> 3; if (deflate_size < min_deflate_size) { return false; } - uint8_t remaining_bit_count = (8 - deflate_bits) & 7; - uint8_t remaining_bits = decInBits.get(remaining_bit_count); - - PreflateParameters params = estimatePreflateParameters(unpacked_output, blocks); - PreflateStatisticsCounter counter; - memset(&counter, 0, sizeof(counter)); - PreflateTokenPredictor tokenPredictor(params, unpacked_output); - PreflateTreePredictor treePredictor(unpacked_output); - for (unsigned i = 0, n = blocks.size(); i < n; ++i) { - tokenPredictor.analyzeBlock(i, blocks[i]); - treePredictor.analyzeBlock(i, blocks[i]); - if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { - return false; - } - tokenPredictor.updateCounters(&counter, i); - treePredictor.updateCounters(&counter, i); - block_callback(); - } - counter.block.incNonZeroPadding(remaining_bits != 0); - PreflateMetaEncoder encoder; - PreflatePredictionEncoder pcodec; - unsigned modelId = encoder.addModel(counter, params); - if (!encoder.beginMetaBlockWithModel(pcodec, modelId)) { - return false; - } - for (unsigned i = 0, n = blocks.size(); i < n; ++i) { - tokenPredictor.encodeBlock(&pcodec, i); - treePredictor.encodeBlock(&pcodec, i); - if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { - return false; - } - tokenPredictor.encodeEOF(&pcodec, i, i + 1 == blocks.size()); - } - pcodec.encodeNonZeroPadding(remaining_bits != 0); - if (remaining_bits != 0) { - unsigned bitsToSave = bitLength(remaining_bits); - pcodec.encodeValue(bitsToSave, 3); - if (bitsToSave > 1) { - pcodec.encodeValue(remaining_bits & ((1 << (bitsToSave - 1)) - 1), bitsToSave - 1); - } - } - if (!encoder.endMetaBlock(pcodec, unpacked_output.size())) { - return false; - } - preflate_diff = encoder.finish(); - return !encoder.error(); + return encoder.finish(preflate_diff); } bool preflate_decode(std::vector& unpacked_output, std::vector& preflate_diff, - const std::vector& deflate_raw) { + uint64_t& deflate_size, + InputStream& deflate_raw, + std::function block_callback, + const size_t min_deflate_size, + const size_t metaBlockSize) { + MemStream uncompressedOutput; + bool result = preflate_decode(uncompressedOutput, preflate_diff, deflate_size, deflate_raw, + block_callback, min_deflate_size, metaBlockSize); + unpacked_output = uncompressedOutput.extractData(); + return result; +} + +bool preflate_decode(std::vector& unpacked_output, + std::vector& preflate_diff, + const std::vector& deflate_raw, + const size_t metaBlockSize) { MemStream mem(deflate_raw); uint64_t raw_size; return preflate_decode(unpacked_output, preflate_diff, - raw_size, mem, [] {}, 0) && raw_size == deflate_raw.size(); + raw_size, mem, [] {}, 0, metaBlockSize) + && raw_size == deflate_raw.size(); } diff --git a/contrib/preflate/preflate_decoder.h b/contrib/preflate/preflate_decoder.h index 3e3ce06..8a462ef 100644 --- a/contrib/preflate/preflate_decoder.h +++ b/contrib/preflate/preflate_decoder.h @@ -17,18 +17,60 @@ #include #include +#include "preflate_statistical_codec.h" #include "preflate_token.h" #include "support/stream.h" +#include "support/task_pool.h" + +class PreflateDecoderTask : public Task { +public: + class Handler { + public: + virtual uint32_t setModel(const PreflateStatisticsCounter&, const PreflateParameters&) = 0; + virtual bool beginEncoding(const uint32_t metaBlockId, PreflatePredictionEncoder&, const uint32_t modelId) = 0; + virtual bool endEncoding(const uint32_t metaBlockId, PreflatePredictionEncoder&, const size_t uncompressedSize) = 0; + virtual void markProgress() = 0; + }; + + PreflateDecoderTask(Handler& handler, + const uint32_t metaBlockId, + std::vector&& tokenData, + std::vector&& uncompressedData, + const size_t uncompressedOffset, + const bool lastMetaBlock, + const uint32_t paddingBits); + + virtual bool execute(); + +private: + Handler& handler; + uint32_t metaBlockId; + std::vector tokenData; + std::vector uncompressedData; + size_t uncompressedOffset; + bool lastMetaBlock; + uint32_t paddingBits; +}; + +bool preflate_decode(OutputStream& unpacked_output, + std::vector& preflate_diff, + uint64_t& deflate_size, + InputStream& deflate_raw, + std::function block_callback, + const size_t min_deflate_size, + const size_t metaBlockSize = INT32_MAX); bool preflate_decode(std::vector& unpacked_output, std::vector& preflate_diff, - const std::vector& deflate_raw); + const std::vector& deflate_raw, + const size_t metaBlockSize = INT32_MAX); bool preflate_decode(std::vector& unpacked_output, std::vector& preflate_diff, uint64_t& deflate_size, InputStream& deflate_raw, std::function block_callback, - const size_t min_deflate_size); + const size_t min_deflate_size, + const size_t metaBlockSize = INT32_MAX); #endif /* PREFLATE_DECODER_H */ diff --git a/contrib/preflate/preflate_parameter_estimator.cpp b/contrib/preflate/preflate_parameter_estimator.cpp index e0548a3..22e3ff5 100644 --- a/contrib/preflate/preflate_parameter_estimator.cpp +++ b/contrib/preflate/preflate_parameter_estimator.cpp @@ -62,6 +62,7 @@ PreflateHuffStrategy estimatePreflateHuffStrategy(const PreflateStreamInfo& info } PreflateParameters estimatePreflateParameters(const std::vector& unpacked_output, + const size_t off0, const std::vector& blocks) { PreflateStreamInfo info = extractPreflateInfo(blocks); @@ -70,7 +71,7 @@ PreflateParameters estimatePreflateParameters(const std::vector& result.memLevel = estimatePreflateMemLevel(info.maxTokensPerBlock); result.strategy = estimatePreflateStrategy(info); result.huffStrategy = estimatePreflateHuffStrategy(info); - PreflateCompLevelInfo cl = estimatePreflateCompLevel(result.windowBits, result.memLevel, unpacked_output, blocks, false); + PreflateCompLevelInfo cl = estimatePreflateCompLevel(result.windowBits, result.memLevel, unpacked_output, off0, blocks, false); result.compLevel = cl.recommendedCompressionLevel; result.zlibCompatible = cl.zlibCompatible; result.farLen3MatchesDetected = cl.farLen3Matches; diff --git a/contrib/preflate/preflate_parameter_estimator.h b/contrib/preflate/preflate_parameter_estimator.h index 177017d..8ebca15 100644 --- a/contrib/preflate/preflate_parameter_estimator.h +++ b/contrib/preflate/preflate_parameter_estimator.h @@ -104,6 +104,7 @@ PreflateHuffStrategy estimatePreflateHuffStrategy(const PreflateStreamInfo&); unsigned char estimatePreflateWindowBits(const unsigned maxDist); PreflateParameters estimatePreflateParameters(const std::vector& unpacked_output, + const size_t off0, const std::vector& blocks); #endif /* PREFLATE_PARAMETER_ESTIMATOR_H */ diff --git a/contrib/preflate/preflate_reencoder.cpp b/contrib/preflate/preflate_reencoder.cpp index 77940a8..b332891 100644 --- a/contrib/preflate/preflate_reencoder.cpp +++ b/contrib/preflate/preflate_reencoder.cpp @@ -21,28 +21,86 @@ #include "support/bitstream.h" #include "support/memstream.h" -bool preflate_reencode(OutputStream& os, - const std::vector& preflate_diff, - const std::vector& unpacked_input, - std::function block_callback) { - PreflateMetaDecoder decoder(preflate_diff, unpacked_input); - if (decoder.error()) { - return false; +class PreflateReencoderHandler : public PreflateReencoderTask::Handler { +public: + PreflateReencoderHandler(BitOutputStream& bos_, + const std::vector& reconData, + const size_t uncompressedSize, + std::function progressCallback_) + : decoder(reconData, uncompressedSize) + , progressCallback(progressCallback_) + , bos(bos_) {} + + size_t metaBlockCount() const { + return decoder.metaBlockCount(); } - if (decoder.metaBlockCount() != 1) { - return false; + size_t metaBlockUncompressedSize(const size_t metaBlockId) const { + return decoder.metaBlockUncompressedSize(metaBlockId); } + bool error() const { + return decoder.error(); + } + + bool finish() { + decoder.finish(); + return !decoder.error(); + } + + virtual bool beginDecoding(const uint32_t metaBlockId, + PreflatePredictionDecoder& codec, PreflateParameters& params) { + return decoder.beginMetaBlock(codec, params, metaBlockId); + } + virtual bool endDecoding(const uint32_t metaBlockId, PreflatePredictionDecoder& codec, + std::vector&& tokenData, + std::vector&& uncompressedData, + const size_t uncompressedOffset, + const size_t paddingBitCount, + const size_t paddingValue) { + if (!decoder.endMetaBlock(codec)) { + return false; + } + + PreflateBlockReencoder deflater(bos, uncompressedData, uncompressedOffset); + for (size_t j = 0, n = tokenData.size(); j < n; ++j) { + deflater.writeBlock(tokenData[j], + metaBlockId + 1 == decoder.metaBlockCount() && j + 1 == n); + markProgress(); + } + bos.put(paddingValue, paddingBitCount); + return true; + } + + virtual void markProgress() { + progressCallback(); + } + +private: + PreflateMetaDecoder decoder; + std::function progressCallback; + BitOutputStream& bos; +}; + +PreflateReencoderTask::PreflateReencoderTask(PreflateReencoderHandler::Handler& handler_, + const uint32_t metaBlockId_, + std::vector&& uncompressedData_, + const size_t uncompressedOffset_, + const bool lastMetaBlock_) + : handler(handler_) + , metaBlockId(metaBlockId_) + , uncompressedData(uncompressedData_) + , uncompressedOffset(uncompressedOffset_) + , lastMetaBlock(lastMetaBlock_) {} + +bool PreflateReencoderTask::execute() { PreflatePredictionDecoder pcodec; PreflateParameters params; - if (!decoder.beginMetaBlock(pcodec, params, 0)) { + if (!handler.beginDecoding(metaBlockId, pcodec, params)) { return false; } - PreflateTokenPredictor tokenPredictor(params, unpacked_input); - PreflateTreePredictor treePredictor(unpacked_input); + PreflateTokenPredictor tokenPredictor(params, uncompressedData, uncompressedOffset); + PreflateTreePredictor treePredictor(uncompressedData, uncompressedOffset); - BitOutputStream bos(os); - - PreflateBlockReencoder deflater(bos, unpacked_input); + std::vector tokenData; bool eof = true; do { PreflateTokenBlock block = tokenPredictor.decodeBlock(&pcodec); @@ -52,25 +110,69 @@ bool preflate_reencode(OutputStream& os, if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { return false; } - eof = tokenPredictor.decodeEOF(&pcodec); - - deflater.writeBlock(block, eof); - block_callback(); - } while (!eof); - bool non_zero_bits = pcodec.decodeNonZeroPadding(); - if (non_zero_bits) { - unsigned bitsToLoad = pcodec.decodeValue(3); - unsigned padding = 0; - if (bitsToLoad > 0) { - padding = (1 << (bitsToLoad - 1)) + pcodec.decodeValue(bitsToLoad - 1); + tokenData.push_back(std::move(block)); + if (!lastMetaBlock) { + eof = tokenPredictor.inputEOF(); + } else { + eof = tokenPredictor.decodeEOF(&pcodec); + } + handler.markProgress(); + } while (!eof); + size_t paddingBitCount = 0; + size_t paddingBits = 0; + if (lastMetaBlock) { + bool non_zero_bits = pcodec.decodeNonZeroPadding(); + if (non_zero_bits) { + paddingBitCount = pcodec.decodeValue(3); + if (paddingBitCount > 0) { + paddingBits = (1 << (paddingBitCount - 1)) + pcodec.decodeValue(paddingBitCount - 1); + } } - bos.put(padding, bitsToLoad); } - if (!decoder.endMetaBlock(pcodec)) { + return handler.endDecoding(metaBlockId, pcodec, std::move(tokenData), + std::move(uncompressedData), uncompressedOffset, + paddingBitCount, paddingBits); +} + +bool preflate_reencode(OutputStream& os, + const std::vector& preflate_diff, + InputStream& is, + const uint64_t unpacked_size, + std::function block_callback) { + BitOutputStream bos(os); + PreflateReencoderHandler decoder(bos, preflate_diff, unpacked_size, block_callback); + if (decoder.error()) { return false; } - deflater.flush(); - return true; + std::vector uncompressedData; + for (size_t j = 0, n = decoder.metaBlockCount(); j < n; ++j) { + size_t curUncSize = uncompressedData.size(); + size_t newSize = decoder.metaBlockUncompressedSize(j); + uncompressedData.resize(curUncSize + newSize); + if (is.read(uncompressedData.data() + curUncSize, newSize) != newSize) { + return false; + } + + PreflateReencoderTask task(decoder, j, std::vector(uncompressedData), curUncSize, j + 1 == n); + if (j + 1 < n) { + uncompressedData.erase(uncompressedData.begin(), + uncompressedData.begin() + std::max(uncompressedData.size(), 1 << 15) - (1 << 15)); + } + + if (!task.execute()) { + return false; + } + } + bos.flush(); + return !decoder.error(); +} + +bool preflate_reencode(OutputStream& os, + const std::vector& preflate_diff, + const std::vector& unpacked_input, + std::function block_callback) { + MemStream is(unpacked_input); + return preflate_reencode(os, preflate_diff, is, unpacked_input.size(), block_callback); } bool preflate_reencode(std::vector& deflate_raw, const std::vector& preflate_diff, diff --git a/contrib/preflate/preflate_reencoder.h b/contrib/preflate/preflate_reencoder.h index 080e22d..3350a8f 100644 --- a/contrib/preflate/preflate_reencoder.h +++ b/contrib/preflate/preflate_reencoder.h @@ -16,11 +16,51 @@ #define PREFLATE_REENCODER_H #include +#include "preflate_statistical_codec.h" +#include +#include + +class PreflateReencoderTask : public Task { +public: + class Handler { + public: + virtual bool beginDecoding(const uint32_t metaBlockId, + PreflatePredictionDecoder&, PreflateParameters&) = 0; + virtual bool endDecoding(const uint32_t metaBlockId, PreflatePredictionDecoder&, + std::vector&& tokenData, + std::vector&& uncompressedData, + const size_t uncompressedOffset, + const size_t paddingBitCount, + const size_t paddingValue) = 0; + virtual void markProgress() = 0; + }; + + PreflateReencoderTask(Handler& handler, + const uint32_t metaBlockId, + std::vector&& uncompressedData, + const size_t uncompressedOffset, + const bool lastMetaBlock); + + virtual bool execute(); + +private: + Handler& handler; + uint32_t metaBlockId; + std::vector uncompressedData; + size_t uncompressedOffset; + bool lastMetaBlock; +}; bool preflate_reencode(std::vector& deflate_raw, const std::vector& preflate_diff, const std::vector& unpacked_input); +bool preflate_reencode(OutputStream& os, + const std::vector& preflate_diff, + InputStream& unpacked_input, + const uint64_t unpacked_size, + std::function block_callback); + bool preflate_reencode(OutputStream& os, const std::vector& preflate_diff, const std::vector& unpacked_input, diff --git a/contrib/preflate/preflate_statistical_codec.cpp b/contrib/preflate/preflate_statistical_codec.cpp index f8d13f3..6b23807 100644 --- a/contrib/preflate/preflate_statistical_codec.cpp +++ b/contrib/preflate/preflate_statistical_codec.cpp @@ -597,9 +597,6 @@ bool PreflateMetaEncoder::endMetaBlock(PreflatePredictionEncoder& encoder, const return true; } std::vector PreflateMetaEncoder::finish() { - if (blockList.size() != 1) { - return std::vector(); - } MemStream mem; BitOutputStream bos(mem); bos.put(0, 1); // no extension used @@ -668,10 +665,10 @@ std::vector PreflateMetaEncoder::finish() { return result; } -PreflateMetaDecoder::PreflateMetaDecoder(const std::vector& reconData_, const std::vector& uncompressed_) +PreflateMetaDecoder::PreflateMetaDecoder(const std::vector& reconData_, const uint64_t uncompressedSize_) : inError(false) , reconData(reconData_) - , uncompressedData(uncompressed_) { + , uncompressedSize(uncompressedSize_) { if (reconData.size() == 0) { inError = true; return; @@ -690,10 +687,6 @@ PreflateMetaDecoder::PreflateMetaDecoder(const std::vector& reconData_, } else { blockCount = 2 + bis.getVLI(); } - if (blockCount != 1) { - inError = true; - return; - } enum Mode { CREATE_NEW_MODEL /*, REUSE_LAST_MODEL, REUSE_PREVIOUS_MODEL*/ }; @@ -769,13 +762,13 @@ PreflateMetaDecoder::PreflateMetaDecoder(const std::vector& reconData_, if (i != blockCount - 1) { reconStart += blockList[i].reconSize; uncStart += blockList[i].uncompressedSize; - if (reconStart > reconData.size() || uncStart > uncompressedData.size()) { + if (reconStart > reconData.size() || uncStart > uncompressedSize) { inError = true; return; } } else { blockList[i].reconSize = reconData.size() - blockList[i].reconStartOfs; - blockList[i].uncompressedSize = uncompressedData.size() - blockList[i].uncompressedStartOfs; + blockList[i].uncompressedSize = uncompressedSize - blockList[i].uncompressedStartOfs; } } } diff --git a/contrib/preflate/preflate_statistical_codec.h b/contrib/preflate/preflate_statistical_codec.h index ba5c0ec..56b3e2f 100644 --- a/contrib/preflate/preflate_statistical_codec.h +++ b/contrib/preflate/preflate_statistical_codec.h @@ -561,7 +561,7 @@ private: }; struct PreflateMetaDecoder { - PreflateMetaDecoder(const std::vector& reconData, const std::vector& uncompressed); + PreflateMetaDecoder(const std::vector& reconData, const uint64_t uncompressedSize); ~PreflateMetaDecoder(); bool error() const { @@ -570,6 +570,13 @@ struct PreflateMetaDecoder { size_t metaBlockCount() const { return blockList.size(); } + uint64_t metaBlockUncompressedStartOfs(const size_t metaBlockId) const { + return blockList[metaBlockId].uncompressedStartOfs; + } + size_t metaBlockUncompressedSize(const size_t metaBlockId) const { + return blockList[metaBlockId].uncompressedSize; + } + bool beginMetaBlock(PreflatePredictionDecoder&, PreflateParameters&, const size_t index); bool endMetaBlock(PreflatePredictionDecoder&); void finish(); @@ -591,8 +598,9 @@ private: bool inError; bool inBlock; + size_t currentMetaBlockId; const std::vector& reconData; - const std::vector& uncompressedData; + const uint64_t uncompressedSize; std::vector modelList; std::vector blockList; }; diff --git a/contrib/preflate/preflate_token_predictor.cpp b/contrib/preflate/preflate_token_predictor.cpp index fbeff42..5f7506e 100644 --- a/contrib/preflate/preflate_token_predictor.cpp +++ b/contrib/preflate/preflate_token_predictor.cpp @@ -20,7 +20,8 @@ PreflateTokenPredictor::PreflateTokenPredictor( const PreflateParameters& params_, - const std::vector& dump) + const std::vector& dump, + const size_t offset) : state(hash, seq, params_.config(), params_.windowBits, params_.memLevel) , hash(dump, params_.memLevel) , seq(dump) @@ -36,6 +37,8 @@ PreflateTokenPredictor::PreflateTokenPredictor( hash.updateRunningHash(state.inputCursor()[1]); seq.updateSeq(2); } + hash.updateHash(offset); + seq.updateSeq(offset); } bool PreflateTokenPredictor::predictEOB() { @@ -494,3 +497,6 @@ bool PreflateTokenPredictor::decodeEOF(PreflatePredictionDecoder* codec) { } return false; } +bool PreflateTokenPredictor::inputEOF() { + return state.availableInputSize() == 0; +} diff --git a/contrib/preflate/preflate_token_predictor.h b/contrib/preflate/preflate_token_predictor.h index 34f5bf2..8d77da7 100644 --- a/contrib/preflate/preflate_token_predictor.h +++ b/contrib/preflate/preflate_token_predictor.h @@ -49,7 +49,8 @@ struct PreflateTokenPredictor { std::vector analysisResults; PreflateTokenPredictor(const PreflateParameters& params, - const std::vector& dump); + const std::vector& uncompressed, + const size_t offset); void analyzeBlock(const unsigned blockno, const PreflateTokenBlock& block); void updateCounters(PreflateStatisticsCounter*, @@ -62,6 +63,7 @@ struct PreflateTokenPredictor { PreflateTokenBlock decodeBlock(PreflatePredictionDecoder*); bool decodeEOF(PreflatePredictionDecoder*); + bool inputEOF(); bool predictEOB(); PreflateToken predictToken(); diff --git a/contrib/preflate/preflate_tree_predictor.cpp b/contrib/preflate/preflate_tree_predictor.cpp index 7082c46..df3c791 100644 --- a/contrib/preflate/preflate_tree_predictor.cpp +++ b/contrib/preflate/preflate_tree_predictor.cpp @@ -20,10 +20,11 @@ #include "preflate_tree_predictor.h" PreflateTreePredictor::PreflateTreePredictor( - const std::vector& dump) + const std::vector& dump, + const size_t off) : input(dump) - , curPos(0) , predictionFailure(false) { + input.advance(off); } struct FreqIdxPair { diff --git a/contrib/preflate/preflate_tree_predictor.h b/contrib/preflate/preflate_tree_predictor.h index 65b15a8..8684234 100644 --- a/contrib/preflate/preflate_tree_predictor.h +++ b/contrib/preflate/preflate_tree_predictor.h @@ -31,7 +31,6 @@ enum TreeCodeType { struct PreflateTreePredictor { PreflateInput input; - unsigned curPos; bool predictionFailure; struct BlockAnalysisResult { @@ -86,7 +85,7 @@ struct PreflateTreePredictor { const unsigned symLCount, const unsigned symDCount); - PreflateTreePredictor(const std::vector& dump); + PreflateTreePredictor(const std::vector& dump, const size_t offset); void analyzeBlock(const unsigned blockno, const PreflateTokenBlock& block); void updateCounters(PreflateStatisticsCounter*, diff --git a/precomp.cpp b/precomp.cpp index 313947a..abb3e68 100644 --- a/precomp.cpp +++ b/precomp.cpp @@ -19,7 +19,7 @@ // version information #define V_MAJOR 0 #define V_MINOR 4 -#define V_MINOR2 131 +#define V_MINOR2 132 //#define V_STATE "ALPHA" #define V_STATE "EXPERIMENTAL (w/ preflate support)" #define V_MSG "USE FOR TESTING ONLY" @@ -54,6 +54,8 @@ #define ERR_ONLY_SET_LZMA_THREAD_ONCE 17 #define ERR_ONLY_SET_LZMA_FILTERS_ONCE 18 +#define NOMINMAX + #include #include #include @@ -208,6 +210,9 @@ bool anything_was_used; bool level_switch_used = false; bool non_zlib_was_used; +// preflate config +size_t meta_block_size = 1 << 21; // 2 MB blocks by default + // statistics unsigned int recompressed_streams_count = 0; unsigned int recompressed_pdf_count = 0; @@ -537,7 +542,7 @@ int init(int argc, char* argv[]) { } printf(" - %s\n",V_MSG); printf("Free for non-commercial use - Copyright 2006-2018 by Christian Schneider\n"); - printf("- experimental preflate v0.2.1 support - Copyright 2018 by Dirk Steinke\n\n"); + printf("- experimental preflate v0.3.1 support - Copyright 2018 by Dirk Steinke\n\n"); // init compression and memory level count bool use_zlib_level[81]; @@ -3095,10 +3100,16 @@ public: return size; } } + _written += size; return own_fwrite(buffer, 1, size, ftempout); } + + uint64_t written() const { + return _written; + } + private: - size_t _written; + uint64_t _written; bool& _in_memory; }; @@ -3113,18 +3124,17 @@ recompress_deflate_result try_recompression_deflate(FILE* file) { memset(&result, 0, sizeof(result)); OwnFileInputStream is(file); - - std::vector unpacked_output; - uint64_t compressed_stream_size = 0; - result.accepted = preflate_decode(unpacked_output, result.recon_data, - compressed_stream_size, is, []() { print_work_sign(true); }, - 0); // you can set a minimum deflate stream size here - result.compressed_stream_size = compressed_stream_size; - result.uncompressed_stream_size = unpacked_output.size(); + { result.uncompressed_in_memory = true; UncompressedOutStream uos(result.uncompressed_in_memory); - uos.write(unpacked_output.data(), unpacked_output.size()); + uint64_t compressed_stream_size = 0; + result.accepted = preflate_decode(uos, result.recon_data, + compressed_stream_size, is, []() { print_work_sign(true); }, + 0, + meta_block_size); // you can set a minimum deflate stream size here + result.compressed_stream_size = compressed_stream_size; + result.uncompressed_stream_size = uos.written(); } return std::move(result); }