denuvo-ref/precomp-cpp
1
0
Fork 0
mirror of https://github.com/schnaader/precomp-cpp synced 2026-08-23 00:23:04 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update to preflate v0.3.1

Browse source 
Splits up large deflate streams to blocks responsible for ~2MiB of
uncompressed data each. This reduces memory pressure a LOT.
This commit is contained in:
Deus Libri 2018-04-30 08:27:18 +02:00
parent 0cdefd8561
commit 4772a9c37b
21 changed files with 550 additions and 178 deletions
Download patch file Download diff file Expand all files Collapse all files

BIN
bin/precomp_w_preflate_msvc17_aka_vc140_x64.exe
View file

Binary file not shown.

BIN
bin/precomp_w_preflate_msvc17_aka_vc140_x86.exe
View file

Binary file not shown.

111
contrib/preflate/README.md
View file

@ -1,4 +1,4 @@
preflate v0.1.2
preflate v0.3.1
===============
Library to split deflate streams into uncompressed data and reconstruction information,
or reconstruct the original deflate stream from those two.
@ -15,19 +15,31 @@ Reconstructing the original deflate stream becomes important if the position or
of the reconstructed deflate streams must not differ, e.g. if those streams are embedded
into executables, or unsupported archive files where the indices cannot be adapted correctly.
There are currently at least two tools available which try to solve this problem:
There are currently already some other tools available which try to solve this problem:
- "precomp" is a tool which can do the bit-correct reconstruction very efficiently,
but only for deflate streams that were created by the zlib library. (It only needs
to store the three relevant zlib parameters to allow reconstruction.)
but only for deflate streams that were created by the ZLIB library. (It only needs
to store the three relevant ZLIB parameters to allow reconstruction.)
It bails out for anything created by 7zip, kzip, or similar tools.
Of course, "precomp" also handles JPG, PNG, ZIP, GIF, PDF, MP3, etc, which makes
a very nice tool, and it is open source.
- "reflate" can reconstruct any deflate stream (also 7zip, kzip, etc), but it is only
efficient for streams that were created by zlib, compression level 9.
All lower compression levels of zlib require increasing reconstruction info, the further
close to perfect for streams that were created by ZLIB, compression level 9.
All lower compression levels of ZLIB require increasing reconstruction info, the further
from level 9, the bigger the required reconstruction data.
"reflate" only handles deflate streams, and is not open source. As far as I know,
it is also part of PowerArchiver.
- "grittibanzli" is also open source, and can reconstruct any deflate stream.
(It was published after the first version of "preflate".)
It outputs the reconstruction data (the diff against the prediction) in
an uncompressed, byte based format, and relies on an external compressor to squeeze it
down. This work quite well, and in my tests, the compressed reconstruction data is
usually only 20 to 30% bigger than "preflate"s, and smaller than "reflate"s.
The ZLIB level detection is rather basic (worse then "preflate"s, and much worse than
"precomp"s), and for streams which were compressed with a lower level of ZLIB, it
will create reconstruction data that are not just a few bytes.
In my tests, compression time was the same for "grittibanzli" and "preflate", while
for decompression it was faster. However, the higher memory requirements will put
a strain on some systems.
What about "difflate"?
@ -40,38 +52,46 @@ development and not yet feature complete. Let's wait and see.
So, what is the point of "preflate"?
------------------------------------
The goal of "preflate" is to get the best of both worlds:
- for deflate streams created by zlib at any compression level, we want to
- for deflate streams created by ZLIB at any compression level, we want to
be able to reconstruct it with only a few bytes of reconstruction information (like "precomp")
- for all other deflate streams, we want to be able to reconstruct them with
reconstruction information that is not much larger than "reflate"
Right now, it has been tested on 11159 valid deflate streams, extracted with
"rawdet" from archives etc., and preflate was capable of inflating and reconstructing
all of them.
Right now, it has been tested on some ten thousand valid deflate streams, some extracted with
"rawdet" from archives etc., and "preflate" was capable of inflating and reconstructing
all of them. There's also an experimental fork of "precomp" which incorporates it,
where it has to cope with lots of invalid deflate streams, or random data which just
looks like a deflate streams. It was tested on several GiB of data with tens of
thousands of valid and invalid deflate streams, and it seems to work quite reliable,
albeit not perfect.
There are still known (and also likely unknown) corner cases of valid deflate stream in
which preflate will fail.
There might be some unknown corner cases of valid deflate stream in which preflate
will fail. And there are probably some cases of invalid deflate streams which will
make it crash.
So, is "preflate" already better than "precomp" and "reflate"?
--------------------------------------------------------------
No. It isn't.
No. It isn't. However, it is a reasonable alternative.
Test coverage is still quite low, and testing it is currently quite cumbersome
(because it only works on raw deflate streams which need to be extracted first.)
First, "preflate" only works on raw deflate streams. So, it is not intended as
a tool, but as a library to be used by some other tool. (Did I mention the "precomp"
fork?) The current ZLIB level detection inside "precomp" is much better than
"preflate"s (and it needs to be), so there are a considerable amount of ZLIB deflate
streams, probably 20-30%, for which "preflate" generates reconstruction data that
is bigger than 3 bytes (usually only a few tens of bytes though).
Or in other words: there are files in which Vanilla "precomp" BEATS "preflate"
in SIZE.
It's very slow. (50-500% slower than "precomp" or "reflate"). Especially for files
containing long runs of the same byte (e.g. \0 or spaces), it gets very, very slow.
It's quite slow. Since 0.2.1, there is some optimization for long runs of the same byte (e.g.
\0 or spaces), and some files profit from that tremendously (which were ten times
slower than now), but for most files the gain is only a few percent.
Both "precomp" and "reflate" BEAT "preflate" in SPEED. (Expected to be around
50-500%).
It will not handle all valid deflate streams. (E.g., preflate will fail if the reference
length 258 is encoded as 227+31.) I don't know if this is really a problem. All good
encoders would never encode a length of 227+31 anyway.
The detection of the zlib compression parameters is not always on spot, which leads to
the creation of larger diffs than necessary.
Right now, it is a proof of concept, that we can do better than both "precomp" and "reflate".
It just isn't stable and fast enough yet to be of practical use.
However, "preflate" eats anything and can be successfully applied to files in which
"precomp" will fail completely. It generates smaller reconstruction data than
"reflate" (in my tests). And it is open source.
How do I build it?
@ -79,15 +99,19 @@ How do I build it?
There is a make file, but it has only been tested so far with MinGW gmake.
The produced executable is larger than 1MiB, while the MSVC compiler generated
executables were around 100KiB. The reason for that is unclear at the moment.
There is also a CMake script which hopefully works for non Windows platforms.
Credits
-------
- "precomp" by Christian Schneider
(https://github.com/schnaader/precomp-cpp)
- "reflate" and "rawdet" by Eugene Shelwien
- "zlib" by Mark Adler et al.
- "7zip" by Igor Pavlov
- "kzip" by Ken Silverman
- "grittibanzli" by Google Zurich (inofficial project)
(https://github.com/google/grittibanzli)
All of the software above is just AWESOME!
@ -102,28 +126,25 @@ Contains information about a lot of interesting compression tools of which I pro
would never have known without this site. Also, Stephan helped getting rid of several
bugs in preflate. Thank you.
- packARI by Matthias Stirner
Notes
-----
Currently, "preflate" uses code from two libraries:
- packARI by Matthias Stirner, which is licensed under LGPL3
packARI provides context-aware, adaptive order(n) encoding. "preflate"s
arithmetic coder is based on packARI's algorithms, but only uses static
models and contains some speed optimizations which are totally pointless because
"preflate" spends all its time in the match finder.
(directory packARI)
It is used for the arithmetic coding of the reconstruction information.
- zlib 1.2.11 by Mark Adler et al., under the zlib license.
(directory zlib 1.2.11.dec. Does NOT contain the full zlib library!)
It is used for the decoding of deflate streams, and some callbacks were
added to get the decoded trees and tokens, which are then used to build
the reconstruction information.
The usage of both libraries will be removed in the future.
There already is a new implementation of deflate decoding (which is still slower
than the zlib one with callbacks).
And packARI is much more powerful and flexible than what is actually needed
in "preflate" right now.
Changes
-------
- 0.1.0 - first public release
- 0.2.0 - freeze bitstream format, remove zlib and packARI dependencies
- 0.2.1 - add match finder for same character sequences
- 0.3.0 - split up large deflate streams into "meta-blocks" of a few megabyte each
(this should help keep memory usage in check)
- 0.3.1 - mitigation against zlib level estimation failure due to
"meta block splitting". For one 350KiB file compressed at zlib level 1,
reconstruction size goes from 35KiB to 1.5KiB. Without meta-blocks it
would be 3 bytes...
License

5
contrib/preflate/preflate_block_reencoder.cpp
View file

@ -20,10 +20,11 @@
PreflateBlockReencoder::PreflateBlockReencoder(
BitOutputStream& bos,
const std::vector<unsigned char>& uncompressedData)
const std::vector<unsigned char>& uncompressedData,
const size_t uncompressedOffset)
: _output(bos)
, _uncompressedData(uncompressedData)
, _uncompressedDataPos(0)
, _uncompressedDataPos(uncompressedOffset)
, _errorCode(OK)
, _dynamicLitLenEncoder(nullptr, 0, false)
, _dynamicDistEncoder(nullptr, 0, false) {

4
contrib/preflate/preflate_block_reencoder.h
View file

@ -46,7 +46,9 @@ public:
const unsigned short *litLenCode, *distCode, *treeCode;
const unsigned char *litLenBits, *distBits, *treeBits;*/
PreflateBlockReencoder(BitOutputStream& bos, const std::vector<unsigned char>& uncompressedData);
PreflateBlockReencoder(BitOutputStream& bos,
const std::vector<unsigned char>& uncompressedData,
const size_t uncompressedOffset);
bool writeBlock(const PreflateTokenBlock&, const bool last);
void flush();

14
contrib/preflate/preflate_checker.cpp
View file

@ -58,7 +58,7 @@ bool preflate_checker(const std::vector<unsigned char>& deflate_raw) {
printf("Unpacked data has size %d\n", (int)unpacked_output.size());
// Encode
PreflateParameters paramsE = estimatePreflateParameters(unpacked_output, blocks);
PreflateParameters paramsE = estimatePreflateParameters(unpacked_output, 0, blocks);
printf("prediction parameters: w %d, c %d, m %d, zlib %d, farL3M %d, very far M %d, M2S %d, log2CD %d\n",
paramsE.windowBits, paramsE.compLevel, paramsE.memLevel,
paramsE.zlibCompatible, paramsE.farLen3MatchesDetected,
@ -67,8 +67,8 @@ bool preflate_checker(const std::vector<unsigned char>& deflate_raw) {
PreflateStatisticsCounter counterE;
memset(&counterE, 0, sizeof(counterE));
PreflateTokenPredictor tokenPredictorE(paramsE, unpacked_output);
PreflateTreePredictor treePredictorE(unpacked_output);
PreflateTokenPredictor tokenPredictorE(paramsE, unpacked_output, 0);
PreflateTreePredictor treePredictorE(unpacked_output, 0);
for (unsigned i = 0, n = blocks.size(); i < n; ++i) {
tokenPredictorE.analyzeBlock(i, blocks[i]);
if (tokenPredictorE.predictionFailure) {
@ -124,7 +124,7 @@ bool preflate_checker(const std::vector<unsigned char>& deflate_raw) {
printf("Prediction diff has size %d\n", (int)preflate_diff.size());
// Decode
PreflateMetaDecoder codecD(preflate_diff, unpacked_output);
PreflateMetaDecoder codecD(preflate_diff, unpacked_output.size());
PreflatePredictionDecoder pcodecD;
PreflateParameters paramsD;
if (codecD.error() || codecD.metaBlockCount() != 1) {
@ -163,13 +163,13 @@ bool preflate_checker(const std::vector<unsigned char>& deflate_raw) {
return false;
}
PreflateTokenPredictor tokenPredictorD(paramsD, unpacked_output);
PreflateTreePredictor treePredictorD(unpacked_output);
PreflateTokenPredictor tokenPredictorD(paramsD, unpacked_output, 0);
PreflateTreePredictor treePredictorD(unpacked_output, 0);
MemStream mem;
BitOutputStream bos(mem);
PreflateBlockReencoder deflater(bos, unpacked_output);
PreflateBlockReencoder deflater(bos, unpacked_output, 0);
unsigned blockno = 0;
bool eof = true;
do {

9
contrib/preflate/preflate_complevel_estimator.cpp
View file

@ -21,6 +21,7 @@ PreflateCompLevelEstimatorState::PreflateCompLevelEstimatorState(
const int wbits,
const int mbits,
const std::vector<unsigned char>& unpacked_output_,
const size_t off0_,
const std::vector<PreflateTokenBlock>& blocks_)
: slowHash(unpacked_output_, mbits)
, fastL1Hash(unpacked_output_, mbits)
@ -28,9 +29,11 @@ PreflateCompLevelEstimatorState::PreflateCompLevelEstimatorState(
, fastL3Hash(unpacked_output_, mbits)
, blocks(blocks_)
, wsize(1 << wbits)
, off0(off0_)
{
memset(&info, 0, sizeof(info));
info.possibleCompressionLevels = (1 << 10) - (1 << 1);
updateHash(off0);
}
void PreflateCompLevelEstimatorState::updateHash(const unsigned len) {
@ -98,6 +101,9 @@ bool PreflateCompLevelEstimatorState::checkMatchSingleFastHash(
return true;
}
void PreflateCompLevelEstimatorState::checkMatch(const PreflateToken& token) {
if (slowHash.input().pos() < token.dist + off0) {
return;
}
unsigned hashHead = slowHash.curHash();
if (info.possibleCompressionLevels & (1 << 1)) {
if (!checkMatchSingleFastHash(token, fastL1Hash, fastPreflateParserSettings[0], hashHead)) {
@ -204,9 +210,10 @@ PreflateCompLevelInfo estimatePreflateCompLevel(
const int wbits,
const int mbits,
const std::vector<unsigned char>& unpacked_output,
const size_t off0,
const std::vector<PreflateTokenBlock>& blocks,
const bool early_out) {
PreflateCompLevelEstimatorState state(wbits, mbits, unpacked_output, blocks);
PreflateCompLevelEstimatorState state(wbits, mbits, unpacked_output, off0, blocks);
state.checkDump(early_out);
state.recommend();
return state.info;

3
contrib/preflate/preflate_complevel_estimator.h
View file

@ -42,9 +42,11 @@ struct PreflateCompLevelEstimatorState {
const std::vector<PreflateTokenBlock>& blocks;
PreflateCompLevelInfo info;
uint16_t wsize;
size_t off0;
PreflateCompLevelEstimatorState(const int wbits, const int mbits,
const std::vector<unsigned char>& unpacked_output,
const size_t off0,
const std::vector<PreflateTokenBlock>& blocks);
void updateHash(const unsigned len);
void updateOrSkipHash(const unsigned len);
@ -67,6 +69,7 @@ PreflateCompLevelInfo estimatePreflateCompLevel(
const int wbits,
const int mbits,
const std::vector<unsigned char>& unpacked_output,
const size_t off0,
const std::vector<PreflateTokenBlock>& blocks,
const bool early_out);

249
contrib/preflate/preflate_decoder.cpp
View file

@ -24,18 +24,110 @@
#include "support/memstream.h"
#include "support/outputcachestream.h"
bool preflate_decode(std::vector<unsigned char>& unpacked_output,
class PreflateDecoderHandler : public PreflateDecoderTask::Handler {
public:
PreflateDecoderHandler(std::function<void(void)> progressCallback_)
: progressCallback(progressCallback_) {}
bool finish(std::vector<uint8_t>& reconstructionData) {
reconstructionData = encoder.finish();
return !encoder.error();
}
bool error() const {
return encoder.error();
}
virtual uint32_t setModel(const PreflateStatisticsCounter& counters, const PreflateParameters& parameters) {
return encoder.addModel(counters, parameters);
}
virtual bool beginEncoding(const uint32_t metaBlockId, PreflatePredictionEncoder& codec, const uint32_t modelId) {
return encoder.beginMetaBlockWithModel(codec, modelId);
}
virtual bool endEncoding(const uint32_t metaBlockId, PreflatePredictionEncoder& codec, const size_t uncompressedSize) {
return encoder.endMetaBlock(codec, uncompressedSize);
}
virtual void markProgress() {
progressCallback();
}
private:
PreflateMetaEncoder encoder;
std::function<void(void)> progressCallback;
};
PreflateDecoderTask::PreflateDecoderTask(PreflateDecoderTask::Handler& handler_,
const uint32_t metaBlockId_,
std::vector<PreflateTokenBlock>&& tokenData_,
std::vector<uint8_t>&& uncompressedData_,
const size_t uncompressedOffset_,
const bool lastMetaBlock_,
const uint32_t paddingBits_)
: handler(handler_)
, metaBlockId(metaBlockId_)
, tokenData(tokenData_)
, uncompressedData(uncompressedData_)
, uncompressedOffset(uncompressedOffset_)
, lastMetaBlock(lastMetaBlock_)
, paddingBits(paddingBits_) {
}
bool PreflateDecoderTask::execute() {
PreflateParameters params = estimatePreflateParameters(uncompressedData, uncompressedOffset, tokenData);
PreflateStatisticsCounter counter;
memset(&counter, 0, sizeof(counter));
PreflateTokenPredictor tokenPredictor(params, uncompressedData, uncompressedOffset);
PreflateTreePredictor treePredictor(uncompressedData, uncompressedOffset);
for (unsigned i = 0, n = tokenData.size(); i < n; ++i) {
tokenPredictor.analyzeBlock(i, tokenData[i]);
treePredictor.analyzeBlock(i, tokenData[i]);
if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) {
return false;
}
tokenPredictor.updateCounters(&counter, i);
treePredictor.updateCounters(&counter, i);
handler.markProgress();
}
counter.block.incNonZeroPadding(paddingBits != 0);
PreflatePredictionEncoder pcodec;
unsigned modelId = handler.setModel(counter, params);
if (!handler.beginEncoding(metaBlockId, pcodec, modelId)) {
return false;
}
for (unsigned i = 0, n = tokenData.size(); i < n; ++i) {
tokenPredictor.encodeBlock(&pcodec, i);
treePredictor.encodeBlock(&pcodec, i);
if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) {
return false;
}
if (lastMetaBlock) {
tokenPredictor.encodeEOF(&pcodec, i, i + 1 == tokenData.size());
}
}
if (lastMetaBlock) {
pcodec.encodeNonZeroPadding(paddingBits != 0);
if (paddingBits != 0) {
unsigned bitsToSave = bitLength(paddingBits);
pcodec.encodeValue(bitsToSave, 3);
if (bitsToSave > 1) {
pcodec.encodeValue(paddingBits & ((1 << (bitsToSave - 1)) - 1), bitsToSave - 1);
}
}
}
return handler.endEncoding(metaBlockId, pcodec, uncompressedData.size() - uncompressedOffset);
}
bool preflate_decode(OutputStream& unpacked_output,
std::vector<unsigned char>& preflate_diff,
uint64_t& deflate_size,
InputStream& deflate_raw,
std::function<void(void)> block_callback,
const size_t min_deflate_size) {
const size_t min_deflate_size,
const size_t metaBlockSize) {
deflate_size = 0;
uint64_t deflate_bits = 0;
size_t prevBitPos = 0;
BitInputStream decInBits(deflate_raw);
MemStream decUnc;
OutputCacheStream decOutCache(decUnc);
OutputCacheStream decOutCache(unpacked_output);
PreflateBlockDecoder bdec(decInBits, decOutCache);
if (bdec.status() != PreflateBlockDecoder::OK) {
return false;
@ -43,6 +135,14 @@ bool preflate_decode(std::vector<unsigned char>& unpacked_output,
bool last;
unsigned i = 0;
std::vector<PreflateTokenBlock> blocks;
std::vector<uint32_t> blockSizes;
uint64_t sumBlockSizes = 0;
uint64_t lastEndPos = 0;
uint64_t uncompressedMetaStart = 0;
size_t MBSize = std::min<size_t>(std::max<size_t>(metaBlockSize, 1 << 18), (1 << 31) - 1);
size_t MBThreshold = (MBSize * 3) >> 1;
PreflateDecoderHandler encoder(block_callback);
size_t MBcount = 0;
do {
PreflateTokenBlock newBlock;
@ -50,74 +150,107 @@ bool preflate_decode(std::vector<unsigned char>& unpacked_output,
if (!ok) {
return false;
}
blocks.push_back(newBlock);
++i;
if (decOutCache.cacheSize() >= 512 * 1024) {
decOutCache.flushUpTo(decOutCache.cacheEndPos() - (32 * 1024));
uint64_t blockSize = decOutCache.cacheEndPos() - lastEndPos;
lastEndPos = decOutCache.cacheEndPos();
if (blockSize >= (1 << 31)) {
// No mega blocks
return false;
}
blocks.push_back(newBlock);
blockSizes.push_back(blockSize);
++i;
block_callback();
deflate_bits += decInBits.bitPos() - prevBitPos;
prevBitPos = decInBits.bitPos();
block_callback();
sumBlockSizes += blockSize;
if (last || sumBlockSizes >= MBThreshold) {
size_t blockCount, blockSizeSum;
if (last) {
blockCount = blockSizes.size();
blockSizeSum = sumBlockSizes;
} else {
blockCount = 0;
blockSizeSum = 0;
for (const auto bs : blockSizes) {
blockSizeSum += bs;
++blockCount;
if (blockSizeSum >= MBSize) {
break;
}
}
}
std::vector<PreflateTokenBlock> blocksForMeta;
for (size_t j = 0; j < blockCount; ++j) {
blocksForMeta.push_back(std::move(blocks[j]));
}
blocks.erase(blocks.begin(), blocks.begin() + blockCount);
blockSizes.erase(blockSizes.begin(), blockSizes.begin() + blockCount);
sumBlockSizes -= blockSizeSum;
size_t uncompressedOffset = MBcount == 0 ? 0 : 1 << 15;
std::vector<uint8_t> uncompressedDataForMeta(
decOutCache.cacheData(uncompressedMetaStart - uncompressedOffset),
decOutCache.cacheData(uncompressedMetaStart - uncompressedOffset) + blockSizeSum + uncompressedOffset);
uncompressedMetaStart += blockSizeSum;
size_t paddingBits = 0;
if (last) {
uint8_t remaining_bit_count = (8 - deflate_bits) & 7;
paddingBits = decInBits.get(remaining_bit_count);
deflate_bits += decInBits.bitPos() - prevBitPos;
prevBitPos = decInBits.bitPos();
}
PreflateDecoderTask task(encoder, MBcount,
std::move(blocksForMeta),
std::move(uncompressedDataForMeta),
uncompressedOffset,
last, paddingBits);
if (!task.execute()) {
return false;
}
MBcount++;
if (!last) {
decOutCache.flushUpTo(uncompressedMetaStart - (1 << 15));
}
}
} while (!last);
decOutCache.flush();
unpacked_output = decUnc.extractData();
deflate_size = (deflate_bits + 7) >> 3;
if (deflate_size < min_deflate_size) {
return false;
}
uint8_t remaining_bit_count = (8 - deflate_bits) & 7;
uint8_t remaining_bits = decInBits.get(remaining_bit_count);
PreflateParameters params = estimatePreflateParameters(unpacked_output, blocks);
PreflateStatisticsCounter counter;
memset(&counter, 0, sizeof(counter));
PreflateTokenPredictor tokenPredictor(params, unpacked_output);
PreflateTreePredictor treePredictor(unpacked_output);
for (unsigned i = 0, n = blocks.size(); i < n; ++i) {
tokenPredictor.analyzeBlock(i, blocks[i]);
treePredictor.analyzeBlock(i, blocks[i]);
if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) {
return false;
}
tokenPredictor.updateCounters(&counter, i);
treePredictor.updateCounters(&counter, i);
block_callback();
}
counter.block.incNonZeroPadding(remaining_bits != 0);
PreflateMetaEncoder encoder;
PreflatePredictionEncoder pcodec;
unsigned modelId = encoder.addModel(counter, params);
if (!encoder.beginMetaBlockWithModel(pcodec, modelId)) {
return false;
}
for (unsigned i = 0, n = blocks.size(); i < n; ++i) {
tokenPredictor.encodeBlock(&pcodec, i);
treePredictor.encodeBlock(&pcodec, i);
if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) {
return false;
}
tokenPredictor.encodeEOF(&pcodec, i, i + 1 == blocks.size());
}
pcodec.encodeNonZeroPadding(remaining_bits != 0);
if (remaining_bits != 0) {
unsigned bitsToSave = bitLength(remaining_bits);
pcodec.encodeValue(bitsToSave, 3);
if (bitsToSave > 1) {
pcodec.encodeValue(remaining_bits & ((1 << (bitsToSave - 1)) - 1), bitsToSave - 1);
}
}
if (!encoder.endMetaBlock(pcodec, unpacked_output.size())) {
return false;
}
preflate_diff = encoder.finish();
return !encoder.error();
return encoder.finish(preflate_diff);
}
bool preflate_decode(std::vector<unsigned char>& unpacked_output,
std::vector<unsigned char>& preflate_diff,
const std::vector<unsigned char>& deflate_raw) {
uint64_t& deflate_size,
InputStream& deflate_raw,
std::function<void(void)> block_callback,
const size_t min_deflate_size,
const size_t metaBlockSize) {
MemStream uncompressedOutput;
bool result = preflate_decode(uncompressedOutput, preflate_diff, deflate_size, deflate_raw,
block_callback, min_deflate_size, metaBlockSize);
unpacked_output = uncompressedOutput.extractData();
return result;
}
bool preflate_decode(std::vector<unsigned char>& unpacked_output,
std::vector<unsigned char>& preflate_diff,
const std::vector<unsigned char>& deflate_raw,
const size_t metaBlockSize) {
MemStream mem(deflate_raw);
uint64_t raw_size;
return preflate_decode(unpacked_output, preflate_diff,
raw_size, mem, [] {}, 0) && raw_size == deflate_raw.size();
raw_size, mem, [] {}, 0, metaBlockSize)
&& raw_size == deflate_raw.size();
}

46
contrib/preflate/preflate_decoder.h
View file

@ -17,18 +17,60 @@
#include <functional>
#include <vector>
#include "preflate_statistical_codec.h"
#include "preflate_token.h"
#include "support/stream.h"
#include "support/task_pool.h"
class PreflateDecoderTask : public Task {
public:
class Handler {
public:
virtual uint32_t setModel(const PreflateStatisticsCounter&, const PreflateParameters&) = 0;
virtual bool beginEncoding(const uint32_t metaBlockId, PreflatePredictionEncoder&, const uint32_t modelId) = 0;
virtual bool endEncoding(const uint32_t metaBlockId, PreflatePredictionEncoder&, const size_t uncompressedSize) = 0;
virtual void markProgress() = 0;
};
PreflateDecoderTask(Handler& handler,
const uint32_t metaBlockId,
std::vector<PreflateTokenBlock>&& tokenData,
std::vector<uint8_t>&& uncompressedData,
const size_t uncompressedOffset,
const bool lastMetaBlock,
const uint32_t paddingBits);
virtual bool execute();
private:
Handler& handler;
uint32_t metaBlockId;
std::vector<PreflateTokenBlock> tokenData;
std::vector<uint8_t> uncompressedData;
size_t uncompressedOffset;
bool lastMetaBlock;
uint32_t paddingBits;
};
bool preflate_decode(OutputStream& unpacked_output,
std::vector<unsigned char>& preflate_diff,
uint64_t& deflate_size,
InputStream& deflate_raw,
std::function<void(void)> block_callback,
const size_t min_deflate_size,
const size_t metaBlockSize = INT32_MAX);
bool preflate_decode(std::vector<unsigned char>& unpacked_output,
std::vector<unsigned char>& preflate_diff,
const std::vector<unsigned char>& deflate_raw);
const std::vector<unsigned char>& deflate_raw,
const size_t metaBlockSize = INT32_MAX);
bool preflate_decode(std::vector<unsigned char>& unpacked_output,
std::vector<unsigned char>& preflate_diff,
uint64_t& deflate_size,
InputStream& deflate_raw,
std::function<void (void)> block_callback,
const size_t min_deflate_size);
const size_t min_deflate_size,
const size_t metaBlockSize = INT32_MAX);
#endif /* PREFLATE_DECODER_H */

3
contrib/preflate/preflate_parameter_estimator.cpp
View file

@ -62,6 +62,7 @@ PreflateHuffStrategy estimatePreflateHuffStrategy(const PreflateStreamInfo& info
}
PreflateParameters estimatePreflateParameters(const std::vector<unsigned char>& unpacked_output,
const size_t off0,
const std::vector<PreflateTokenBlock>& blocks) {
PreflateStreamInfo info = extractPreflateInfo(blocks);
@ -70,7 +71,7 @@ PreflateParameters estimatePreflateParameters(const std::vector<unsigned char>&
result.memLevel = estimatePreflateMemLevel(info.maxTokensPerBlock);
result.strategy = estimatePreflateStrategy(info);
result.huffStrategy = estimatePreflateHuffStrategy(info);
PreflateCompLevelInfo cl = estimatePreflateCompLevel(result.windowBits, result.memLevel, unpacked_output, blocks, false);
PreflateCompLevelInfo cl = estimatePreflateCompLevel(result.windowBits, result.memLevel, unpacked_output, off0, blocks, false);
result.compLevel = cl.recommendedCompressionLevel;
result.zlibCompatible = cl.zlibCompatible;
result.farLen3MatchesDetected = cl.farLen3Matches;

1
contrib/preflate/preflate_parameter_estimator.h
View file

@ -104,6 +104,7 @@ PreflateHuffStrategy estimatePreflateHuffStrategy(const PreflateStreamInfo&);
unsigned char estimatePreflateWindowBits(const unsigned maxDist);
PreflateParameters estimatePreflateParameters(const std::vector<unsigned char>& unpacked_output,
const size_t off0,
const std::vector<PreflateTokenBlock>& blocks);
#endif /* PREFLATE_PARAMETER_ESTIMATOR_H */

162
contrib/preflate/preflate_reencoder.cpp
View file

@ -21,28 +21,86 @@
#include "support/bitstream.h"
#include "support/memstream.h"
bool preflate_reencode(OutputStream& os,
const std::vector<unsigned char>& preflate_diff,
const std::vector<unsigned char>& unpacked_input,
std::function<void(void)> block_callback) {
PreflateMetaDecoder decoder(preflate_diff, unpacked_input);
if (decoder.error()) {
return false;
class PreflateReencoderHandler : public PreflateReencoderTask::Handler {
public:
PreflateReencoderHandler(BitOutputStream& bos_,
const std::vector<uint8_t>& reconData,
const size_t uncompressedSize,
std::function<void(void)> progressCallback_)
: decoder(reconData, uncompressedSize)
, progressCallback(progressCallback_)
, bos(bos_) {}
size_t metaBlockCount() const {
return decoder.metaBlockCount();
}
if (decoder.metaBlockCount() != 1) {
return false;
size_t metaBlockUncompressedSize(const size_t metaBlockId) const {
return decoder.metaBlockUncompressedSize(metaBlockId);
}
bool error() const {
return decoder.error();
}
bool finish() {
decoder.finish();
return !decoder.error();
}
virtual bool beginDecoding(const uint32_t metaBlockId,
PreflatePredictionDecoder& codec, PreflateParameters& params) {
return decoder.beginMetaBlock(codec, params, metaBlockId);
}
virtual bool endDecoding(const uint32_t metaBlockId, PreflatePredictionDecoder& codec,
std::vector<PreflateTokenBlock>&& tokenData,
std::vector<uint8_t>&& uncompressedData,
const size_t uncompressedOffset,
const size_t paddingBitCount,
const size_t paddingValue) {
if (!decoder.endMetaBlock(codec)) {
return false;
}
PreflateBlockReencoder deflater(bos, uncompressedData, uncompressedOffset);
for (size_t j = 0, n = tokenData.size(); j < n; ++j) {
deflater.writeBlock(tokenData[j],
metaBlockId + 1 == decoder.metaBlockCount() && j + 1 == n);
markProgress();
}
bos.put(paddingValue, paddingBitCount);
return true;
}
virtual void markProgress() {
progressCallback();
}
private:
PreflateMetaDecoder decoder;
std::function<void(void)> progressCallback;
BitOutputStream& bos;
};
PreflateReencoderTask::PreflateReencoderTask(PreflateReencoderHandler::Handler& handler_,
const uint32_t metaBlockId_,
std::vector<uint8_t>&& uncompressedData_,
const size_t uncompressedOffset_,
const bool lastMetaBlock_)
: handler(handler_)
, metaBlockId(metaBlockId_)
, uncompressedData(uncompressedData_)
, uncompressedOffset(uncompressedOffset_)
, lastMetaBlock(lastMetaBlock_) {}
bool PreflateReencoderTask::execute() {
PreflatePredictionDecoder pcodec;
PreflateParameters params;
if (!decoder.beginMetaBlock(pcodec, params, 0)) {
if (!handler.beginDecoding(metaBlockId, pcodec, params)) {
return false;
}
PreflateTokenPredictor tokenPredictor(params, unpacked_input);
PreflateTreePredictor treePredictor(unpacked_input);
PreflateTokenPredictor tokenPredictor(params, uncompressedData, uncompressedOffset);
PreflateTreePredictor treePredictor(uncompressedData, uncompressedOffset);
BitOutputStream bos(os);
PreflateBlockReencoder deflater(bos, unpacked_input);
std::vector<PreflateTokenBlock> tokenData;
bool eof = true;
do {
PreflateTokenBlock block = tokenPredictor.decodeBlock(&pcodec);
@ -52,25 +110,69 @@ bool preflate_reencode(OutputStream& os,
if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) {
return false;
}
eof = tokenPredictor.decodeEOF(&pcodec);
deflater.writeBlock(block, eof);
block_callback();
} while (!eof);
bool non_zero_bits = pcodec.decodeNonZeroPadding();
if (non_zero_bits) {
unsigned bitsToLoad = pcodec.decodeValue(3);
unsigned padding = 0;
if (bitsToLoad > 0) {
padding = (1 << (bitsToLoad - 1)) + pcodec.decodeValue(bitsToLoad - 1);
tokenData.push_back(std::move(block));
if (!lastMetaBlock) {
eof = tokenPredictor.inputEOF();
} else {
eof = tokenPredictor.decodeEOF(&pcodec);
}
handler.markProgress();
} while (!eof);
size_t paddingBitCount = 0;
size_t paddingBits = 0;
if (lastMetaBlock) {
bool non_zero_bits = pcodec.decodeNonZeroPadding();
if (non_zero_bits) {
paddingBitCount = pcodec.decodeValue(3);
if (paddingBitCount > 0) {
paddingBits = (1 << (paddingBitCount - 1)) + pcodec.decodeValue(paddingBitCount - 1);
}
}
bos.put(padding, bitsToLoad);
}
if (!decoder.endMetaBlock(pcodec)) {
return handler.endDecoding(metaBlockId, pcodec, std::move(tokenData),
std::move(uncompressedData), uncompressedOffset,
paddingBitCount, paddingBits);
}
bool preflate_reencode(OutputStream& os,
const std::vector<unsigned char>& preflate_diff,
InputStream& is,
const uint64_t unpacked_size,
std::function<void(void)> block_callback) {
BitOutputStream bos(os);
PreflateReencoderHandler decoder(bos, preflate_diff, unpacked_size, block_callback);
if (decoder.error()) {
return false;
}
deflater.flush();
return true;
std::vector<uint8_t> uncompressedData;
for (size_t j = 0, n = decoder.metaBlockCount(); j < n; ++j) {
size_t curUncSize = uncompressedData.size();
size_t newSize = decoder.metaBlockUncompressedSize(j);
uncompressedData.resize(curUncSize + newSize);
if (is.read(uncompressedData.data() + curUncSize, newSize) != newSize) {
return false;
}
PreflateReencoderTask task(decoder, j, std::vector<uint8_t>(uncompressedData), curUncSize, j + 1 == n);
if (j + 1 < n) {
uncompressedData.erase(uncompressedData.begin(),
uncompressedData.begin() + std::max<size_t>(uncompressedData.size(), 1 << 15) - (1 << 15));
}
if (!task.execute()) {
return false;
}
}
bos.flush();
return !decoder.error();
}
bool preflate_reencode(OutputStream& os,
const std::vector<unsigned char>& preflate_diff,
const std::vector<unsigned char>& unpacked_input,
std::function<void(void)> block_callback) {
MemStream is(unpacked_input);
return preflate_reencode(os, preflate_diff, is, unpacked_input.size(), block_callback);
}
bool preflate_reencode(std::vector<unsigned char>& deflate_raw,
const std::vector<unsigned char>& preflate_diff,

40
contrib/preflate/preflate_reencoder.h
View file

@ -16,11 +16,51 @@
#define PREFLATE_REENCODER_H
#include <vector>
#include "preflate_statistical_codec.h"
#include <support/stream.h>
#include <support/task_pool.h>
class PreflateReencoderTask : public Task {
public:
class Handler {
public:
virtual bool beginDecoding(const uint32_t metaBlockId,
PreflatePredictionDecoder&, PreflateParameters&) = 0;
virtual bool endDecoding(const uint32_t metaBlockId, PreflatePredictionDecoder&,
std::vector<PreflateTokenBlock>&& tokenData,
std::vector<uint8_t>&& uncompressedData,
const size_t uncompressedOffset,
const size_t paddingBitCount,
const size_t paddingValue) = 0;
virtual void markProgress() = 0;
};
PreflateReencoderTask(Handler& handler,
const uint32_t metaBlockId,
std::vector<uint8_t>&& uncompressedData,
const size_t uncompressedOffset,
const bool lastMetaBlock);
virtual bool execute();
private:
Handler& handler;
uint32_t metaBlockId;
std::vector<uint8_t> uncompressedData;
size_t uncompressedOffset;
bool lastMetaBlock;
};
bool preflate_reencode(std::vector<unsigned char>& deflate_raw,
const std::vector<unsigned char>& preflate_diff,
const std::vector<unsigned char>& unpacked_input);
bool preflate_reencode(OutputStream& os,
const std::vector<unsigned char>& preflate_diff,
InputStream& unpacked_input,
const uint64_t unpacked_size,
std::function<void(void)> block_callback);
bool preflate_reencode(OutputStream& os,
const std::vector<unsigned char>& preflate_diff,
const std::vector<unsigned char>& unpacked_input,

15
contrib/preflate/preflate_statistical_codec.cpp
View file

@ -597,9 +597,6 @@ bool PreflateMetaEncoder::endMetaBlock(PreflatePredictionEncoder& encoder, const
return true;
}
std::vector<unsigned char> PreflateMetaEncoder::finish() {
if (blockList.size() != 1) {
return std::vector<unsigned char>();
}
MemStream mem;
BitOutputStream bos(mem);
bos.put(0, 1); // no extension used
@ -668,10 +665,10 @@ std::vector<unsigned char> PreflateMetaEncoder::finish() {
return result;
}
PreflateMetaDecoder::PreflateMetaDecoder(const std::vector<uint8_t>& reconData_, const std::vector<uint8_t>& uncompressed_)
PreflateMetaDecoder::PreflateMetaDecoder(const std::vector<uint8_t>& reconData_, const uint64_t uncompressedSize_)
: inError(false)
, reconData(reconData_)
, uncompressedData(uncompressed_) {
, uncompressedSize(uncompressedSize_) {
if (reconData.size() == 0) {
inError = true;
return;
@ -690,10 +687,6 @@ PreflateMetaDecoder::PreflateMetaDecoder(const std::vector<uint8_t>& reconData_,
} else {
blockCount = 2 + bis.getVLI();
}
if (blockCount != 1) {
inError = true;
return;
}
enum Mode {
CREATE_NEW_MODEL /*, REUSE_LAST_MODEL, REUSE_PREVIOUS_MODEL*/
};
@ -769,13 +762,13 @@ PreflateMetaDecoder::PreflateMetaDecoder(const std::vector<uint8_t>& reconData_,
if (i != blockCount - 1) {
reconStart += blockList[i].reconSize;
uncStart += blockList[i].uncompressedSize;
if (reconStart > reconData.size() || uncStart > uncompressedData.size()) {
if (reconStart > reconData.size() || uncStart > uncompressedSize) {
inError = true;
return;
}
} else {
blockList[i].reconSize = reconData.size() - blockList[i].reconStartOfs;
blockList[i].uncompressedSize = uncompressedData.size() - blockList[i].uncompressedStartOfs;
blockList[i].uncompressedSize = uncompressedSize - blockList[i].uncompressedStartOfs;
}
}
}

12
contrib/preflate/preflate_statistical_codec.h
View file

@ -561,7 +561,7 @@ private:
};
struct PreflateMetaDecoder {
PreflateMetaDecoder(const std::vector<uint8_t>& reconData, const std::vector<uint8_t>& uncompressed);
PreflateMetaDecoder(const std::vector<uint8_t>& reconData, const uint64_t uncompressedSize);
~PreflateMetaDecoder();
bool error() const {
@ -570,6 +570,13 @@ struct PreflateMetaDecoder {
size_t metaBlockCount() const {
return blockList.size();
}
uint64_t metaBlockUncompressedStartOfs(const size_t metaBlockId) const {
return blockList[metaBlockId].uncompressedStartOfs;
}
size_t metaBlockUncompressedSize(const size_t metaBlockId) const {
return blockList[metaBlockId].uncompressedSize;
}
bool beginMetaBlock(PreflatePredictionDecoder&, PreflateParameters&, const size_t index);
bool endMetaBlock(PreflatePredictionDecoder&);
void finish();
@ -591,8 +598,9 @@ private:
bool inError;
bool inBlock;
size_t currentMetaBlockId;
const std::vector<uint8_t>& reconData;
const std::vector<uint8_t>& uncompressedData;
const uint64_t uncompressedSize;
std::vector<modelType> modelList;
std::vector<metaBlockInfo> blockList;
};

8
contrib/preflate/preflate_token_predictor.cpp
View file

@ -20,7 +20,8 @@
PreflateTokenPredictor::PreflateTokenPredictor(
const PreflateParameters& params_,
const std::vector<unsigned char>& dump)
const std::vector<unsigned char>& dump,
const size_t offset)
: state(hash, seq, params_.config(), params_.windowBits, params_.memLevel)
, hash(dump, params_.memLevel)
, seq(dump)
@ -36,6 +37,8 @@ PreflateTokenPredictor::PreflateTokenPredictor(
hash.updateRunningHash(state.inputCursor()[1]);
seq.updateSeq(2);
}
hash.updateHash(offset);
seq.updateSeq(offset);
}
bool PreflateTokenPredictor::predictEOB() {
@ -494,3 +497,6 @@ bool PreflateTokenPredictor::decodeEOF(PreflatePredictionDecoder* codec) {
}
return false;
}
bool PreflateTokenPredictor::inputEOF() {
return state.availableInputSize() == 0;
}

4
contrib/preflate/preflate_token_predictor.h
View file

@ -49,7 +49,8 @@ struct PreflateTokenPredictor {
std::vector<BlockAnalysisResult> analysisResults;
PreflateTokenPredictor(const PreflateParameters& params,
const std::vector<unsigned char>& dump);
const std::vector<unsigned char>& uncompressed,
const size_t offset);
void analyzeBlock(const unsigned blockno,
const PreflateTokenBlock& block);
void updateCounters(PreflateStatisticsCounter*,
@ -62,6 +63,7 @@ struct PreflateTokenPredictor {
PreflateTokenBlock decodeBlock(PreflatePredictionDecoder*);
bool decodeEOF(PreflatePredictionDecoder*);
bool inputEOF();
bool predictEOB();
PreflateToken predictToken();

5
contrib/preflate/preflate_tree_predictor.cpp
View file

@ -20,10 +20,11 @@
#include "preflate_tree_predictor.h"
PreflateTreePredictor::PreflateTreePredictor(
const std::vector<unsigned char>& dump)
const std::vector<unsigned char>& dump,
const size_t off)
: input(dump)
, curPos(0)
, predictionFailure(false) {
input.advance(off);
}
struct FreqIdxPair {

3
contrib/preflate/preflate_tree_predictor.h
View file

@ -31,7 +31,6 @@ enum TreeCodeType {
struct PreflateTreePredictor {
PreflateInput input;
unsigned curPos;
bool predictionFailure;
struct BlockAnalysisResult {
@ -86,7 +85,7 @@ struct PreflateTreePredictor {
const unsigned symLCount,
const unsigned symDCount);
PreflateTreePredictor(const std::vector<unsigned char>& dump);
PreflateTreePredictor(const std::vector<unsigned char>& dump, const size_t offset);
void analyzeBlock(const unsigned blockno,
const PreflateTokenBlock& block);
void updateCounters(PreflateStatisticsCounter*,

34
precomp.cpp
View file

@ -19,7 +19,7 @@
// version information
#define V_MAJOR 0
#define V_MINOR 4
#define V_MINOR2 131
#define V_MINOR2 132
//#define V_STATE "ALPHA"
#define V_STATE "EXPERIMENTAL (w/ preflate support)"
#define V_MSG "USE FOR TESTING ONLY"
@ -54,6 +54,8 @@
#define ERR_ONLY_SET_LZMA_THREAD_ONCE 17
#define ERR_ONLY_SET_LZMA_FILTERS_ONCE 18
#define NOMINMAX
#include <stdio.h>
#include <iostream>
#include <string.h>
@ -208,6 +210,9 @@ bool anything_was_used;
bool level_switch_used = false;
bool non_zlib_was_used;
// preflate config
size_t meta_block_size = 1 << 21; // 2 MB blocks by default
// statistics
unsigned int recompressed_streams_count = 0;
unsigned int recompressed_pdf_count = 0;
@ -537,7 +542,7 @@ int init(int argc, char* argv[]) {
}
printf(" - %s\n",V_MSG);
printf("Free for non-commercial use - Copyright 2006-2018 by Christian Schneider\n");
printf("- experimental preflate v0.2.1 support - Copyright 2018 by Dirk Steinke\n\n");
printf("- experimental preflate v0.3.1 support - Copyright 2018 by Dirk Steinke\n\n");
// init compression and memory level count
bool use_zlib_level[81];
@ -3095,10 +3100,16 @@ public:
return size;
}
}
_written += size;
return own_fwrite(buffer, 1, size, ftempout);
}
uint64_t written() const {
return _written;
}
private:
size_t _written;
uint64_t _written;
bool& _in_memory;
};
@ -3113,18 +3124,17 @@ recompress_deflate_result try_recompression_deflate(FILE* file) {
memset(&result, 0, sizeof(result));
OwnFileInputStream is(file);
std::vector<unsigned char> unpacked_output;
uint64_t compressed_stream_size = 0;
result.accepted = preflate_decode(unpacked_output, result.recon_data,
compressed_stream_size, is, []() { print_work_sign(true); },
0); // you can set a minimum deflate stream size here
result.compressed_stream_size = compressed_stream_size;
result.uncompressed_stream_size = unpacked_output.size();
{
result.uncompressed_in_memory = true;
UncompressedOutStream uos(result.uncompressed_in_memory);
uos.write(unpacked_output.data(), unpacked_output.size());
uint64_t compressed_stream_size = 0;
result.accepted = preflate_decode(uos, result.recon_data,
compressed_stream_size, is, []() { print_work_sign(true); },
0,
meta_block_size); // you can set a minimum deflate stream size here
result.compressed_stream_size = compressed_stream_size;
result.uncompressed_stream_size = uos.written();
}
return std::move(result);
}