diff --git a/README.md b/README.md index 62b43a0..23822e1 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,20 @@ Precomp [![Packaging status](https://repology.org/badge/vertical-allrepos/precomp.svg)](https://repology.org/metapackage/precomp) +Fork note +--------- +This is an experimental fork of precomp, that uses the preflate library for +deflate stream recompression. +The bitstream format output by the preflate library is not stable yet, +so do not use this experimental tool for archiving. + +Also, the build system isn't up to date yet. I'll provide a CMake script +in the near future. Also, importing the preflate sources in a GUI, e.g. MSVC, +is pretty straight forward. + +For the time being, there are executables for testing. + + What is Precomp? ---------------- Precomp is a command line precompressor. You can use it to achieve better compression on some file-/streamtypes (works on files and streams that are compressed with zLib or the Deflate compression method, bZip2, GIF, JPG and MP3). Precomp tries to decompress the streams, and if they can be decompressed and "re-"compressed so that they are bit-to-bit-identical with the original stream, the decompressed stream can be used instead of the compressed one. diff --git a/bin/precomp_w_preflate_msvc17_aka_vc140_x64.exe b/bin/precomp_w_preflate_msvc17_aka_vc140_x64.exe new file mode 100644 index 0000000..6f6cb15 Binary files /dev/null and b/bin/precomp_w_preflate_msvc17_aka_vc140_x64.exe differ diff --git a/bin/precomp_w_preflate_msvc17_aka_vc140_x86.exe b/bin/precomp_w_preflate_msvc17_aka_vc140_x86.exe new file mode 100644 index 0000000..e81d19c Binary files /dev/null and b/bin/precomp_w_preflate_msvc17_aka_vc140_x86.exe differ diff --git a/contrib/preflate/LICENSE b/contrib/preflate/LICENSE new file mode 100644 index 0000000..261eeb9 --- /dev/null +++ b/contrib/preflate/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/contrib/preflate/Makefile b/contrib/preflate/Makefile new file mode 100644 index 0000000..c024264 --- /dev/null +++ b/contrib/preflate/Makefile @@ -0,0 +1,63 @@ +CC = g++ +COPT = -x c -std=c99 -O3 -funroll-loops -ffast-math -fomit-frame-pointer -DZ_SOLO -DNO_GZIP +CCOPT = -std=c++11 -O3 -Wall -Wno-misleading-indentation -pedantic -funroll-loops -ffast-math -fomit-frame-pointer +DEMONAME = preflate_demo.exe +OBJDIR = objs +BINDIR = bin + +PREFLATE_LIB_FILEROOTS = block_decoder block_reencoder block_trees complevel_estimator \ + constants decoder hash_chain info parameter_estimator parser_config \ + predictor_state reencoder statistical_codec statistical_model \ + token_predictor token tree_predictor +SUPPORT_LIB_FILEROOTS = arithmetic_coder array_helper bit_helper bitstream const_division \ + filestream huffman_decoder huffman_encoder huffman_helper \ + memstream outputcachestream support_tests task_pool +PACKARI_FILEROOTS = aricoder bitops +PREFLATE_DEMO_FILEROOTS = main preflate_checker preflate_dumper preflate_unpack +ZLIB_FILEROOTS = adler32 inffast inflate inftrees trees zutil +DEMO_FILEROOTS = main checker dumper unpack + +PREFLATE_LIB_SRC_FILES = $(addprefix preflate_, $(addsuffix .cpp, $(PREFLATE_LIB_FILEROOTS))) +PREFLATE_LIB_OBJ_FILES = $(addprefix $(OBJDIR)/preflate_, $(addsuffix .o, $(PREFLATE_LIB_FILEROOTS))) +SUPPORT_LIB_SRC_FILES = $(addprefix support/, $(addsuffix .cpp, $(SUPPORT_LIB_FILEROOTS))) +SUPPORT_LIB_OBJ_FILES = $(addprefix $(OBJDIR)/prefsup_, $(addsuffix .o, $(SUPPORT_LIB_FILEROOTS))) +PACKARI_SRC_FILES = $(addprefix packARI/, $(addsuffix .cpp, $(PACKARI_FILEROOTS))) +PACKARI_OBJ_FILES = $(addprefix $(OBJDIR)/packari_, $(addsuffix .o, $(PACKARI_FILEROOTS))) +ZLIB_SRC_FILES = $(addprefix zlib1.2.11.dec/, $(addsuffix .cpp, $(ZLIB_FILEROOTS))) +ZLIB_OBJ_FILES = $(addprefix $(OBJDIR)/zlib_, $(addsuffix .o, $(ZLIB_FILEROOTS))) +DEMO_OBJ_FILES = $(addprefix $(OBJDIR)/prefdemo_, $(addsuffix .o, $(DEMO_FILEROOTS))) + +.PHONY: all +all: dirs bin/$(DEMONAME) + +.PHONY: clean +clean: + rm -f objs/*.o + +dirs: objs bin lib + +objs bin lib: + mkdir $@ + +lib/libpreflate.a: $(PREFLATE_LIB_OBJ_FILES) $(SUPPORT_LIB_OBJ_FILES) $(PACKARI_OBJ_FILES) + ar r $@ $? + +bin/$(DEMONAME): $(ZLIB_OBJ_FILES) $(DEMO_OBJ_FILES) lib/libpreflate.a + $(CC) $(CCOPT) -s -o $@ $? + +$(OBJDIR)/preflate_%.o : preflate_%.cpp + $(CC) $(CCOPT) -Werror -o $@ -c -Isupport -IpackARI $< +$(OBJDIR)/prefsup_%.o : support/%.cpp + $(CC) $(CCOPT) -Werror -o $@ -c -Isupport -IpackARI $< +$(OBJDIR)/packari_%.o : packARI/source/%.cpp + $(CC) $(CCOPT) -o $@ -c $< +$(OBJDIR)/zlib_%.o : zlib1.2.11.dec/%.c + $(CC) $(COPT) -o $@ -c -Izlib1.2.13.dec $< +$(OBJDIR)/prefdemo_main.o : main.cpp + $(CC) $(CCOPT) -Werror -o $@ -c -Isupport -IpackARI -Izlib1.2.13.dec $< +$(OBJDIR)/prefdemo_checker.o : preflate_checker.cpp + $(CC) $(CCOPT) -Werror -o $@ -c -Isupport $< +$(OBJDIR)/prefdemo_dumper.o : preflate_dumper.cpp + $(CC) $(CCOPT) -Werror -o $@ -c -Isupport -Izlib1.2.13.dec $< +$(OBJDIR)/prefdemo_unpack.o : preflate_unpack.cpp + $(CC) $(CCOPT) -Werror -o $@ -c -Isupport -Izlib1.2.13.dec $< diff --git a/contrib/preflate/README.md b/contrib/preflate/README.md new file mode 100644 index 0000000..3ffcca9 --- /dev/null +++ b/contrib/preflate/README.md @@ -0,0 +1,143 @@ +preflate v0.1.2 +=============== +Library to split deflate streams into uncompressed data and reconstruction information, +or reconstruct the original deflate stream from those two. + + +What's it all about? +-------------------- +Long term storage. There are a lot of documents, archives, etc that contain deflate stream. +Deflate is a very simple, but not very strong compression algorithm. For long term storage, +it makes sense to compress the uncompressed deflate stream with a stronger algorithm, like +lzma, rolz, bwt or others. + +Reconstructing the original deflate stream becomes important if the position or size +of the reconstructed deflate streams must not differ, e.g. if those streams are embedded +into executables, or unsupported archive files where the indices cannot be adapted correctly. + +There are currently at least two tools available which try to solve this problem: +- "precomp" is a tool which can do the bit-correct reconstruction very efficiently, + but only for deflate streams that were created by the zlib library. (It only needs + to store the three relevant zlib parameters to allow reconstruction.) + It bails out for anything created by 7zip, kzip, or similar tools. + Of course, "precomp" also handles JPG, PNG, ZIP, GIF, PDF, MP3, etc, which makes + a very nice tool, and it is open source. +- "reflate" can reconstruct any deflate stream (also 7zip, kzip, etc), but it is only + efficient for streams that were created by zlib, compression level 9. + All lower compression levels of zlib require increasing reconstruction info, the further + from level 9, the bigger the required reconstruction data. + "reflate" only handles deflate streams, and is not open source. As far as I know, + it is also part of PowerArchiver. + + +What about "difflate"? +---------------------- +The author of "precomp" has announced quite some time ago to work on "difflate", +which should basically be an open source alternative to "reflate". It is still in +development and not yet feature complete. Let's wait and see. + + +So, what is the point of "preflate"? +------------------------------------ +The goal of "preflate" is to get the best of both worlds: +- for deflate streams created by zlib at any compression level, we want to + be able to reconstruct it with only a few bytes of reconstruction information (like "precomp") +- for all other deflate streams, we want to be able to reconstruct them with + reconstruction information that is not much larger than "reflate" + +Right now, it has been tested on 11159 valid deflate streams, extracted with +"rawdet" from archives etc., and preflate was capable of inflating and reconstructing +all of them. + +There are still known (and also likely unknown) corner cases of valid deflate stream in +which preflate will fail. + + +So, is "preflate" already better than "precomp" and "reflate"? +-------------------------------------------------------------- +No. It isn't. + +Test coverage is still quite low, and testing it is currently quite cumbersome +(because it only works on raw deflate streams which need to be extracted first.) + +It's very slow. (50-500% slower than "precomp" or "reflate"). Especially for files +containing long runs of the same byte (e.g. \0 or spaces), it gets very, very slow. + +It will not handle all valid deflate streams. (E.g., preflate will fail if the reference +length 258 is encoded as 227+31.) I don't know if this is really a problem. All good +encoders would never encode a length of 227+31 anyway. + +The detection of the zlib compression parameters is not always on spot, which leads to +the creation of larger diffs than necessary. + +Right now, it is a proof of concept, that we can do better than both "precomp" and "reflate". +It just isn't stable and fast enough yet to be of practical use. + + +How do I build it? +------------------ +There is a make file, but it has only been tested so far with MinGW gmake. +The produced executable is larger than 1MiB, while the MSVC compiler generated +executables were around 100KiB. The reason for that is unclear at the moment. + + +Credits +------- +- "precomp" by Christian Schneider +- "reflate" and "rawdet" by Eugene Shelwien +- "zlib" by Mark Adler et al. +- "7zip" by Igor Pavlov +- "kzip" by Ken Silverman + +All of the software above is just AWESOME! + +- the ENCODE.RU data compression forum (http://encode.ru/forum/2-Data-Compression) + +If you want to know about the new hot stuff in data compression (doesn't happen +too often though), look here first. + +- www.squeezechart.com by Stephan Busch + +Contains information about a lot of interesting compression tools of which I probably +would never have known without this site. Also, Stephan helped getting rid of several +bugs in preflate. Thank you. + + +Notes +----- +Currently, "preflate" uses code from two libraries: +- packARI by Matthias Stirner, which is licensed under LGPL3 + + (directory packARI) + + It is used for the arithmetic coding of the reconstruction information. +- zlib 1.2.11 by Mark Adler et al., under the zlib license. + + (directory zlib 1.2.11.dec. Does NOT contain the full zlib library!) + + It is used for the decoding of deflate streams, and some callbacks were + added to get the decoded trees and tokens, which are then used to build + the reconstruction information. + +The usage of both libraries will be removed in the future. +There already is a new implementation of deflate decoding (which is still slower +than the zlib one with callbacks). +And packARI is much more powerful and flexible than what is actually needed +in "preflate" right now. + + +License +------- +Copyright 2018 Dirk Steinke + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. diff --git a/contrib/preflate/main.cpp b/contrib/preflate/main.cpp new file mode 100644 index 0000000..4aa2cd9 --- /dev/null +++ b/contrib/preflate/main.cpp @@ -0,0 +1,178 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include +#include +#include +#include + +#include "preflate_checker.h" +#include "preflate_decoder.h" +#include "preflate_info.h" +#include "preflate_reencoder.h" +#include "support/support_tests.h" + +bool loadfile( + std::vector& content, + const std::string& fn) { + FILE* f = fopen(fn.c_str(), "rb"); + if (!f) { + return false; + } + fseek(f, 0, SEEK_END); + long length = ftell(f); + fseek(f, 0, SEEK_SET); + content.resize(length); + long read = fread(content.data(), 1, content.size(), f); + fclose(f); + return read == length; +} +bool savefile( + const std::vector& content, + const std::string& fn) { + FILE* f = fopen(fn.c_str(), "wb"); + if (!f) { + return false; + } + size_t written = fwrite(content.data(), 1, content.size(), f); + fclose(f); + return written == content.size(); +} + +int test(const char* const * const fns, const unsigned fncnt) { + bool ok = true; + for (unsigned i = 0; i < fncnt; ++i) { + std::vector content; + if (!loadfile(content, fns[i])) { + printf("loading of %s failed\n", fns[i]); + ok = false; + } else { + printf("checking %s\n", fns[i]); + bool check_ok = preflate_checker(content); + ok = ok && check_ok; + } + } + if (ok) { + printf("All checks ok\n"); + } + return ok ? 0 : -1; +} +int split(const char* const * const fns, const unsigned fncnt) { + bool ok = true; + for (unsigned i = 0; i < fncnt; ++i) { + std::vector content; + if (!loadfile(content, fns[i])) { + printf("loading of %s failed\n", fns[i]); + ok = false; + } else { + std::vector unpacked; + std::vector recon; + bool check_ok = preflate_decode(unpacked, recon, content); + if (check_ok) { + savefile(unpacked, std::string(fns[i]) + ".u"); + savefile(recon, std::string(fns[i]) + ".r"); + printf("splitting %s successful (%d -> %d + %d)\n", + fns[i], (int)content.size(), (int)unpacked.size(), (int)recon.size()); + } else { + printf("splitting %s failed\n", fns[i]); + } + ok = ok && check_ok; + } + } + if (ok) { + printf("All ok\n"); + } + return ok ? 0 : -1; +} +int combine(const char* const * const fns, const unsigned fncnt, const std::string& ext) { + bool ok = true; + for (unsigned i = 0; i < fncnt; ++i) { + std::vector unpacked; + std::vector recon; + if (!loadfile(unpacked, std::string(fns[i]) + ".u") + || !loadfile(recon, std::string(fns[i]) + ".r")) { + printf("loading of %s.u/.r failed\n", fns[i]); + ok = false; + } else { + std::vector content; + bool check_ok = preflate_reencode(content, recon, unpacked); + if (check_ok) { + savefile(content, std::string(fns[i]) + ext); + printf("recombining %s%s successful (%d + %d -> %d)\n", + fns[i], ext.c_str(), (int)unpacked.size(), (int)recon.size(), (int)content.size()); + } else { + printf("recombining %s%s failed\n", fns[i], ext.c_str()); + } + ok = ok && check_ok; + } + } + if (ok) { + printf("All ok\n"); + } + return ok ? 0 : -1; +} + +int main(int argc, const char * const * const argv) { + if (!support_self_tests()) { + return -1; + } + +/* static const char* const def_fns[] = { + "../testdata/zlib1.raw", "../testdata/zlib2.raw", "../testdata/zlib3.raw", + "../testdata/zlib4.raw", "../testdata/zlib5.raw", "../testdata/zlib6.raw", + "../testdata/zlib7.raw", "../testdata/zlib9.raw", + "../testdata/7zfast.raw", "../testdata/7zultra.raw", "../testdata/kz.raw", + "../testdata/dump.raw", "../testdata/dump_kz.raw", + "../testdata/new_pet.raw", "../testdata/bluevi6.raw", + "../testdata/test1a.png.raw", "../testdata/test1b.png.raw", "../testdata/test2b.png.raw", + }; + */ +static const char* const def_fns[] = { + "../testdata/broken/00000065_inf.raw", +}; + const char* const * fns = def_fns; + unsigned fncnt = sizeof(def_fns) / sizeof(def_fns[0]); + + if (argc >= 2) { + if (!strcmp(argv[1], "-t")) { + if (argc >= 3) { + fns = argv + 2; + fncnt = argc - 2; + } + return test(fns, fncnt); + } + if (argc >= 3 && !strcmp(argv[1], "-s")) { + return split(argv + 2, argc - 2); + } + if (argc >= 3 && !strcmp(argv[1], "-r")) { + return combine(argv + 2, argc - 2, ""); + } + if (argc >= 3 && !strcmp(argv[1], "-x")) { + return combine(argv + 2, argc - 2, ".x"); + } + } + printf("usage: %s -t FILE [FILE ... FILE]\n", argv[0]); + printf(" test uncompression and recompression\n"); + printf(" %s -s FILE [FILE ... FILE]\n", argv[0]); + printf(" split deflate stream (FILE) into uncompressed part\n"); + printf(" (FILE.u) and reconstruction info (FILE.r)\n"); + printf(" %s -r FILE [FILE ... FILE]\n", argv[0]); + printf(" recombine uncompressed part (FILE.u) and reconstruction\n"); + printf(" info (FILE.r) into deflate stream (FILE)\n"); + printf(" %s -x FILE [FILE ... FILE]\n", argv[0]); + printf(" recombines into FILE.x instead of FILE\n"); + return -1; +} diff --git a/contrib/preflate/packARI/.gitignore b/contrib/preflate/packARI/.gitignore new file mode 100644 index 0000000..7c4e65b --- /dev/null +++ b/contrib/preflate/packARI/.gitignore @@ -0,0 +1,31 @@ +# Compiled Object files +*.slo +*.lo +*.o +*.obj + +# Precompiled Headers +*.gch +*.pch + +# Compiled Dynamic libraries +*.so +*.dylib +*.dll + +# Fortran module files +*.mod + +# Compiled Static libraries +*.lai +*.la +*.a +*.lib + +# Executables +*.exe +*.out +*.app + +# Folders to ignore +/zzz_backup/ diff --git a/contrib/preflate/packARI/LICENSE b/contrib/preflate/packARI/LICENSE new file mode 100644 index 0000000..65c5ca8 --- /dev/null +++ b/contrib/preflate/packARI/LICENSE @@ -0,0 +1,165 @@ + GNU LESSER GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + + This version of the GNU Lesser General Public License incorporates +the terms and conditions of version 3 of the GNU General Public +License, supplemented by the additional permissions listed below. + + 0. Additional Definitions. + + As used herein, "this License" refers to version 3 of the GNU Lesser +General Public License, and the "GNU GPL" refers to version 3 of the GNU +General Public License. + + "The Library" refers to a covered work governed by this License, +other than an Application or a Combined Work as defined below. + + An "Application" is any work that makes use of an interface provided +by the Library, but which is not otherwise based on the Library. +Defining a subclass of a class defined by the Library is deemed a mode +of using an interface provided by the Library. + + A "Combined Work" is a work produced by combining or linking an +Application with the Library. The particular version of the Library +with which the Combined Work was made is also called the "Linked +Version". + + The "Minimal Corresponding Source" for a Combined Work means the +Corresponding Source for the Combined Work, excluding any source code +for portions of the Combined Work that, considered in isolation, are +based on the Application, and not on the Linked Version. + + The "Corresponding Application Code" for a Combined Work means the +object code and/or source code for the Application, including any data +and utility programs needed for reproducing the Combined Work from the +Application, but excluding the System Libraries of the Combined Work. + + 1. Exception to Section 3 of the GNU GPL. + + You may convey a covered work under sections 3 and 4 of this License +without being bound by section 3 of the GNU GPL. + + 2. Conveying Modified Versions. + + If you modify a copy of the Library, and, in your modifications, a +facility refers to a function or data to be supplied by an Application +that uses the facility (other than as an argument passed when the +facility is invoked), then you may convey a copy of the modified +version: + + a) under this License, provided that you make a good faith effort to + ensure that, in the event an Application does not supply the + function or data, the facility still operates, and performs + whatever part of its purpose remains meaningful, or + + b) under the GNU GPL, with none of the additional permissions of + this License applicable to that copy. + + 3. Object Code Incorporating Material from Library Header Files. + + The object code form of an Application may incorporate material from +a header file that is part of the Library. You may convey such object +code under terms of your choice, provided that, if the incorporated +material is not limited to numerical parameters, data structure +layouts and accessors, or small macros, inline functions and templates +(ten or fewer lines in length), you do both of the following: + + a) Give prominent notice with each copy of the object code that the + Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the object code with a copy of the GNU GPL and this license + document. + + 4. Combined Works. + + You may convey a Combined Work under terms of your choice that, +taken together, effectively do not restrict modification of the +portions of the Library contained in the Combined Work and reverse +engineering for debugging such modifications, if you also do each of +the following: + + a) Give prominent notice with each copy of the Combined Work that + the Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the Combined Work with a copy of the GNU GPL and this license + document. + + c) For a Combined Work that displays copyright notices during + execution, include the copyright notice for the Library among + these notices, as well as a reference directing the user to the + copies of the GNU GPL and this license document. + + d) Do one of the following: + + 0) Convey the Minimal Corresponding Source under the terms of this + License, and the Corresponding Application Code in a form + suitable for, and under terms that permit, the user to + recombine or relink the Application with a modified version of + the Linked Version to produce a modified Combined Work, in the + manner specified by section 6 of the GNU GPL for conveying + Corresponding Source. + + 1) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (a) uses at run time + a copy of the Library already present on the user's computer + system, and (b) will operate properly with a modified version + of the Library that is interface-compatible with the Linked + Version. + + e) Provide Installation Information, but only if you would otherwise + be required to provide such information under section 6 of the + GNU GPL, and only to the extent that such information is + necessary to install and execute a modified version of the + Combined Work produced by recombining or relinking the + Application with a modified version of the Linked Version. (If + you use option 4d0, the Installation Information must accompany + the Minimal Corresponding Source and Corresponding Application + Code. If you use option 4d1, you must provide the Installation + Information in the manner specified by section 6 of the GNU GPL + for conveying Corresponding Source.) + + 5. Combined Libraries. + + You may place library facilities that are a work based on the +Library side by side in a single library together with other library +facilities that are not Applications and are not covered by this +License, and convey such a combined library under terms of your +choice, if you do both of the following: + + a) Accompany the combined library with a copy of the same work based + on the Library, uncombined with any other library facilities, + conveyed under the terms of this License. + + b) Give prominent notice with the combined library that part of it + is a work based on the Library, and explaining where to find the + accompanying uncombined form of the same work. + + 6. Revised Versions of the GNU Lesser General Public License. + + The Free Software Foundation may publish revised and/or new versions +of the GNU Lesser General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. + + Each version is given a distinguishing version number. If the +Library as you received it specifies that a certain numbered version +of the GNU Lesser General Public License "or any later version" +applies to it, you have the option of following the terms and +conditions either of that published version or of any later version +published by the Free Software Foundation. If the Library as you +received it does not specify a version number of the GNU Lesser +General Public License, you may choose any version of the GNU Lesser +General Public License ever published by the Free Software Foundation. + + If the Library as you received it specifies that a proxy can decide +whether future versions of the GNU Lesser General Public License shall +apply, that proxy's public statement of acceptance of any version is +permanent authorization for you to choose that version for the +Library. diff --git a/contrib/preflate/packARI/README.md b/contrib/preflate/packARI/README.md new file mode 100644 index 0000000..e4c6954 --- /dev/null +++ b/contrib/preflate/packARI/README.md @@ -0,0 +1,8 @@ +# packARI +A very basic compression tool for any file, mainly intended as base for bigger projects + +packARI is a compression tool based solely on the core entropy encoding component used in all my other compression softwares (that means packJPG, packMP3, packPNM, …). On it’s own it might not achieve good results for some file types, but it makes for a great starting point when developing compression tools of your own. + +The entropy coding in packARI has been tested since the beginning in all my other compression softwares, so it might be the most stable of them all. + +You may freely use packARI and it’s source code under the terms of the LGPL. diff --git a/contrib/preflate/packARI/source/Makefile b/contrib/preflate/packARI/source/Makefile new file mode 100644 index 0000000..fb7b98e --- /dev/null +++ b/contrib/preflate/packARI/source/Makefile @@ -0,0 +1,56 @@ +# packXXX Makefile, based on UniMake: Universal Makefile +# Created by Matthias Stirner, 01/2016 + +TARGET = packARI +CC = gcc +CPP = g++ +RC = windres -O coff +CPPFLAGS = -I. -O3 -Wall -pedantic -funroll-loops -ffast-math -fsched-spec-load -fomit-frame-pointer +LDFLAGS = -static -static-libgcc -static-libstdc++ +CSRC = $(wildcard *.c) +CPPSRC = $(wildcard *.cpp) +DEPS = $(wildcard *.h) Makefile +OBJ = $(patsubst %.c,%.o,$(CSRC)) $(patsubst %.cpp,%.o,$(CPPSRC)) + +# conditional stuff +ifeq ($(OS),Windows_NT) +LDFLAGS += -lpthread -L libwinpthread-1.dll +RES = icons.res +UPX := -upx --best --lzma $(TARGET).exe +else +CPPFLAGS += -DUNIX +RC = +RES = +UPX = +endif + +%.o: %.cpp $(DEPS) + $(CPP) -c -o $@ $< $(CPPFLAGS) + +%.res: %.rc + @-$(RC) $< $@ + +$(TARGET): $(OBJ) $(RES) + $(CPP) -o $@ $^ -s $(LDFLAGS) + $(UPX) + +.PHONY: all dev lib dll + +all: $(TARGET) + +dev: CPPFLAGS += -DDEV_BUILD +dev: $(TARGET) + +lib: CPPFLAGS += -DBUILD_LIB +lib: $(OBJ) + ar r $(TARGET)lib.a $(OBJ) + ranlib $(TARGET)lib.a + +dll: CPPFLAGS += -DBUILD_DLL +dll: LDFLAGS += -Wl,--out-implib,libpackJPG.a -fvisibility=hidden +dll: $(OBJ) + $(CPP) -shared -o $(TARGET).dll $^ $(LDFLAGS) + +clean: + @echo clean... + @-rm *.o *.a $(TARGET) $(TARGET).exe $(TARGET).dll diff --git a/contrib/preflate/packARI/source/app_icon.ico b/contrib/preflate/packARI/source/app_icon.ico new file mode 100644 index 0000000..dcb2008 Binary files /dev/null and b/contrib/preflate/packARI/source/app_icon.ico differ diff --git a/contrib/preflate/packARI/source/aricoder.cpp b/contrib/preflate/packARI/source/aricoder.cpp new file mode 100644 index 0000000..eb8a596 --- /dev/null +++ b/contrib/preflate/packARI/source/aricoder.cpp @@ -0,0 +1,1044 @@ +#include +#include "bitops.h" +#include "aricoder.h" + +#define ERROR_EXIT { error = true; exit( 0 ); } + + +/* ----------------------------------------------- + constructor for aricoder class + ----------------------------------------------- */ + +aricoder::aricoder( iostream* stream, int iomode ) +{ + // iomode (i/o mode) + // 0 -> reading + // 1 -> writing + + int i; + + // set initial values + ccode = 0; + clow = 0; + chigh = CODER_LIMIT100 - 1; + cstep = 0; + bbyte = 0; + cbit = 0; + nrbits = 0; + + // store pointer to iostream for reading/writing + sptr = stream; + + // store i/o mode + mode = iomode; + + if ( mode == 0 ) { // mode is reading / decoding + // code buffer has to be filled before starting decoding + for ( i = 0; i < CODER_USE_BITS; i++ ) + ccode = ( ccode << 1 ) | read_bit(); + } // mode is writing / encoding otherwise +} + +/* ----------------------------------------------- + destructor for aricoder class + ----------------------------------------------- */ + +aricoder::~aricoder( void ) +{ + if ( mode == 1 ) { // mode is writing / encoding + // due to clow < CODER_LIMIT050, and chigh >= CODER_LIMIT050 + // there are only two possible cases + if ( clow < CODER_LIMIT025 ) { // case a.) + write_bit( 0 ); + // write remaining bits + write_bit( 1 ); + while ( nrbits-- > 0 ) + write_bit( 1 ); + } + else { // case b.), clow >= CODER_LIMIT025 + write_bit( 1 ); + } // done, zeroes are auto-read by the decoder + + // pad code with zeroes + while ( cbit > 0 ) write_bit( 0 ); + } +} + +/* ----------------------------------------------- + arithmetic encoder function + ----------------------------------------------- */ + +void aricoder::encode( symbol* s ) +{ + // update steps, low count, high count + cstep = ( ( chigh - clow ) + 1 ) / s->scale; + chigh = clow + ( cstep * s->high_count ) - 1; + clow = clow + ( cstep * s->low_count ); + + // e3 scaling is performed for speed and to avoid underflows + // if both, low and high are either in the lower half or in the higher half + // one bit can be safely shifted out + while ( ( clow >= CODER_LIMIT050 ) || ( chigh < CODER_LIMIT050 ) ) { + if ( chigh < CODER_LIMIT050 ) { // this means both, high and low are below, and 0 can be safely shifted out + // write 0 bit + write_bit( 0 ); + // shift out remaing e3 bits + for ( ; nrbits > 0; nrbits-- ) + write_bit( 1 ); + } + else { // if the first wasn't the case, it's clow >= CODER_LIMIT050 + // write 1 bit + write_bit( 1 ); + clow &= CODER_LIMIT050 - 1; + chigh &= CODER_LIMIT050 - 1; + // shift out remaing e3 bits + for ( ; nrbits > 0; nrbits-- ) + write_bit( 0 ); + } + clow <<= 1; + chigh <<= 1; + chigh++; + } + + // e3 scaling, to make sure that theres enough space between low and high + while ( ( clow >= CODER_LIMIT025 ) && ( chigh < CODER_LIMIT075 ) ) { + nrbits++; + clow &= CODER_LIMIT025 - 1; + chigh ^= CODER_LIMIT025 + CODER_LIMIT050; + // clow -= CODER_LIMIT025; + // chigh -= CODER_LIMIT025; + clow <<= 1; + chigh <<= 1; + chigh++; + } +} + +/* ----------------------------------------------- + arithmetic decoder get count function + ----------------------------------------------- */ + +unsigned int aricoder::decode_count( symbol* s ) +{ + // update cstep, which is needed to remove the symbol from the stream later + cstep = ( ( chigh - clow ) + 1 ) / s->scale; + + // return counts, needed to decode the symbol from the statistical model + return ( ccode - clow ) / cstep; +} + +/* ----------------------------------------------- + arithmetic decoder function + ----------------------------------------------- */ + +void aricoder::decode( symbol* s ) +{ + // no actual decoding takes place, as this has to happen in the statistical model + // the symbol has to be removed from the stream, though + + // alread have steps updated from decoder_count + // update low count and high count + chigh = clow + ( cstep * s->high_count ) - 1; + clow = clow + ( cstep * s->low_count ); + + // e3 scaling is performed for speed and to avoid underflows + // if both, low and high are either in the lower half or in the higher half + // one bit can be safely shifted out + while ( ( clow >= CODER_LIMIT050 ) || ( chigh < CODER_LIMIT050 ) ) { + if ( clow >= CODER_LIMIT050 ) { + clow &= CODER_LIMIT050 - 1; + chigh &= CODER_LIMIT050 - 1; + ccode &= CODER_LIMIT050 - 1; + } // if the first wasn't the case, it's chigh < CODER_LIMIT050 + clow <<= 1; + chigh <<= 1; + chigh++; + ccode <<= 1; + ccode |= read_bit(); + nrbits = 0; + } + + // e3 scaling, to make sure that theres enough space between low and high + while ( ( clow >= CODER_LIMIT025 ) && ( chigh < CODER_LIMIT075 ) ) { + nrbits++; + clow &= CODER_LIMIT025 - 1; + chigh ^= CODER_LIMIT025 + CODER_LIMIT050; + // clow -= CODER_LIMIT025; + // chigh -= CODER_LIMIT025; + ccode -= CODER_LIMIT025; + clow <<= 1; + chigh <<= 1; + chigh++; + ccode <<= 1; + ccode |= read_bit(); + } +} + +/* ----------------------------------------------- + bit writer function + ----------------------------------------------- */ + +void aricoder::write_bit( unsigned char bit ) +{ + // add bit at last position + bbyte = ( bbyte << 1 ) | bit; + // increment bit position + cbit++; + + // write bit if done + if ( cbit == 8 ) { + sptr->write( (void*) &bbyte, 1, 1 ); + cbit = 0; + } +} + +/* ----------------------------------------------- + bit reader function + ----------------------------------------------- */ + +unsigned char aricoder::read_bit( void ) +{ + // read in new byte if needed + if ( cbit == 0 ) { + if ( sptr->read( &bbyte, 1, 1 ) == 0 ) // read next byte if available + bbyte = 0; // if no more data is left in the stream + cbit = 8; + } + + // decrement current bit position + cbit--; + // return bit at cbit position + return BITN( bbyte, cbit ); +} + + +/* ----------------------------------------------- + universal statistical model for arithmetic coding + ----------------------------------------------- */ + +model_s::model_s( int max_s, int max_c, int max_o, int c_lim ) +{ + // boundaries of this model: + // max_s (maximum symbol) -> 1 <= max_s <= 1024 (???) + // max_c (maximum context) -> 1 <= max_c <= 1024 (???) + // max_o (maximum order) -> -1 <= max_o <= 4 + // c_lim (maximum count) -> 2 <= c_lim <= 4096 (???) + // WARNING: this can be memory intensive, so don't overdo it + // max_s == 256; max_c == 256; max_o == 4 would be way too much + + table_s* null_table; + table_s* start_table; + int i; + + + // set error false + error = false; + + // copy settings into model + max_symbol = max_s; + max_context = max_c; + max_order = max_o; + max_count = c_lim; + + + // alloc memory for totals table + // totals = ( unsigned short* ) calloc( max_symbol + 2, sizeof( short ) ); + totals = ( unsigned int* ) calloc( max_symbol + 2, sizeof( int ) ); + + // alloc memory for scoreboard, set sb0_count + scoreboard = ( char* ) calloc( max_symbol, sizeof( char ) ); + sb0_count = max_symbol; + + // set current order + current_order = max_order; + + + // set up null table + null_table = ( table_s* ) calloc( 1, sizeof( table_s ) ); + if ( null_table == NULL ) ERROR_EXIT; + null_table->counts = ( unsigned short* ) calloc( max_symbol, sizeof( short ) ); + if ( null_table->counts == NULL ) ERROR_EXIT; + for ( i = 0; i < max_symbol; i++ ) + null_table->counts[ i ] = 1; // set all probabilities + // set up internal counts + null_table->max_count = 1; + null_table->max_symbol = max_symbol; + + // set up start table + start_table = ( table_s* ) calloc( 1, sizeof( table_s ) ); + if ( start_table == NULL ) ERROR_EXIT; + start_table->links = ( table_s** ) calloc( max_context, sizeof( table_s* ) ); + if ( start_table->links == NULL ) ERROR_EXIT; + // set up internal counts + start_table->max_count = 0; + start_table->max_symbol = 0; + + // build links for start table & null table + start_table->lesser = null_table; + null_table->links = ( table_s** ) calloc( max_context, sizeof( table_s* ) ); + if ( null_table->links == NULL ) ERROR_EXIT; + for ( i = 0; i < max_context; i++ ) + null_table->links[ i ] = start_table; + + // alloc memory for storage & contexts + storage = ( table_s** ) calloc( max_order + 3, sizeof( table_s* ) ); + if ( storage == NULL ) ERROR_EXIT; + contexts = storage + 1; + + // integrate tables into contexts + contexts[ -1 ] = null_table; + contexts[ 0 ] = start_table; + + // build initial 'normal' tables + for ( i = 1; i <= max_order; i++ ) { + // set up current order table + contexts[ i ] = ( table_s* ) calloc( 1, sizeof( table_s ) ); + if ( contexts[ i ] == NULL ) ERROR_EXIT; + contexts[ i ]->max_count = 0; + contexts[ i ]->max_symbol = 0; + // build forward and backward links + contexts[ i ]->lesser = contexts[ i - 1 ]; + if ( i < max_order ) { + contexts[ i ]->links = ( table_s** ) calloc( max_context, sizeof( table_s* ) ); + if ( contexts[ i ]->links == NULL ) ERROR_EXIT; + } + else { + contexts[ i ]->links = NULL; + } + contexts[ i - 1 ]->links[ 0 ] = contexts[ i ]; + } +} + + +/* ----------------------------------------------- + model class destructor - recursive cleanup of memory is done here + ----------------------------------------------- */ + +model_s::~model_s( void ) +{ + table_s* context; + + + // clean up each 'normal' table + context = contexts[ 0 ]; + recursive_cleanup ( context ); + + // clean up null table + context = contexts[ -1 ]; + if ( context->links != NULL ) + free( context->links ); + if ( context->counts != NULL ) free( context->counts ); + free ( context ); + + // free everything else + free( storage ); + free( totals ); + free( scoreboard ); +} + + +/* ----------------------------------------------- + updates statistics for a specific symbol / resets to highest order + ----------------------------------------------- */ + +void model_s::update_model( int symbol ) +{ + // use -1 if you just want to reset without updating statistics + + table_s* context; + unsigned short* counts; + int local_order; + int i; + + + // only contexts, that were actually used to encode + // the symbol get their counts updated + if ( symbol >= 0 ) { + for ( local_order = ( current_order < 0 ) ? 0 : current_order; + local_order <= max_order; local_order++ ) { + context = contexts[ local_order ]; + counts = context->counts + symbol; + // update count for specific symbol & scale + (*counts)++; + // store side information for totalize_table + if ( (*counts) > context->max_count ) context->max_count = (*counts); + if ( symbol >= context->max_symbol ) context->max_symbol = symbol+1; + // if counts for that symbol have gone above the maximum count + // the table has to be resized (scale factor 2) + if ( (*counts) >= max_count ) + rescale_table( context, 1 ); + } + } + + // reset scoreboard and current order + current_order = max_order; + for ( i = 0; i < max_symbol; i++ ) + scoreboard[ i ] = 0; + sb0_count = max_symbol; +} + + +/* ----------------------------------------------- + shift in one context (max no of contexts is max_c) + ----------------------------------------------- */ + +void model_s::shift_context( int c ) +{ + table_s* context; + int i; + + // shifting is not possible if max_order is below 1 + // or context index is negative + if ( ( max_order < 1 ) || ( c < 0 ) ) return; + + // shift each orders' context + for ( i = max_order; i > 0; i-- ) { + // this is the new current order context + context = contexts[ i - 1 ]->links[ c ]; + + // check if context exists, build if needed + if ( context == NULL ) { + // reserve memory for next table_s + context = ( table_s* ) calloc( 1, sizeof( table_s ) ); + if ( context == NULL ) ERROR_EXIT; + // set counts NULL + context->counts = NULL; + // setup internal counts + context->max_count = 0; + context->max_symbol = 0; + // link lesser context later if not existing, this is done below + context->lesser = contexts[ i - 2 ]->links[ c ]; + // finished here if this is a max order context + if ( i == max_order ) + context->links = NULL; + else { + // build links to higher order tables otherwise + context->links = ( table_s** ) calloc( max_context, sizeof( table_s* ) ); + if ( context->links == NULL ) ERROR_EXIT; + // add lesser link for higher context (see above) + contexts[ i + 1 ]->lesser = context; + } + // put context to its right place + contexts[ i - 1 ]->links[ c ] = context; + } + + // switch context + contexts[ i ] = context; + } +} + + +/* ----------------------------------------------- + flushes the whole model by diviging through a specific scale factor + ----------------------------------------------- */ + +void model_s::flush_model( int scale_factor ) +{ + recursive_flush( contexts[ 0 ], scale_factor ); +} + + +/* ----------------------------------------------- + exclude specific symbols using this function + ----------------------------------------------- */ + +void model_s::exclude_symbols( char rule, int c ) +{ + // exclusions are back to normal after update_model is used + // modify scoreboard according to rule and value + switch ( rule ) + { + case 'a': + // above rule + // every symbol above c is excluded + for ( c = c + 1; c < max_symbol; c++ ) { + if ( scoreboard[ c ] == 0 ) { + scoreboard[ c ] = 1; + sb0_count--; + } + } + break; + + case 'b': + // below rule + // every symbol below c is excluded + for ( c = c - 1; c >= 0; c-- ) { + if ( scoreboard[ c ] == 0 ) { + scoreboard[ c ] = 1; + sb0_count--; + } + } + break; + + case 'e': + // equal rule + // only c is excluded + if ( scoreboard[ c ] == 0 ) { + scoreboard[ c ] = 1; + sb0_count--; + } + break; + + default: + // unknown rule + // do nothing + break; + } +} + + +/* ----------------------------------------------- + converts an int to a symbol, needed only when encoding + ----------------------------------------------- */ + +int model_s::convert_int_to_symbol( int c, symbol *s ) +{ + // search the symbol c in the current context table_s, + // return scale, low- and high counts + + table_s* context; + + + // totalize table for the current context + context = contexts[ current_order ]; + totalize_table( context ); + + // finding the scale is easy + s->scale = totals[ 0 ]; + + // check if that symbol exists in the current table. send escape otherwise + if ( c >= 0 ) { + if ( context->counts[ c ] > 0 ) { + // return high and low count for the current symbol + s->low_count = totals[ c + 2 ]; + s->high_count = totals[ c + 1 ]; + return 0; + } + } + + // return high and low count for the escape symbol + s->low_count = totals[ 1 ]; + s->high_count = totals[ 0 ]; + current_order--; + return 1; +} + + +/* ----------------------------------------------- + returns the current context scale needed only when decoding + ----------------------------------------------- */ + +void model_s::get_symbol_scale( symbol *s ) +{ + // getting the scale is easy: totalize the table_s, use accumulated count -> done + totalize_table( contexts[ current_order ] ); + s->scale = totals[ 0 ]; +} + + +/* ----------------------------------------------- + converts a count to an int, called after get_symbol_scale + ----------------------------------------------- */ + +int model_s::convert_symbol_to_int( int count, symbol *s ) +{ + // seek the symbol that matches the count, + // also, set low- and high count for the symbol - it has to be removed from the stream + + int c; + + // go through the totals table, search the symbol that matches the count + for ( c = 1; count < (signed) totals[ c ]; c++ ); + // set up the current symbol + s->low_count = totals[ c ]; + s->high_count = totals[ c - 1 ]; + // send escape if escape symbol encountered + if ( c == 1 ) { + current_order--; + return ESCAPE_SYMBOL; + } + + // return symbol value + return ( c - 2 ); +} + + +/* ----------------------------------------------- + totals are calculated by accumulating counts in the current table_s + ----------------------------------------------- */ + +void model_s::totalize_table( table_s *context ) +{ + // update exclusion is used, so this has to be done each time + // escape probability calculation also takes place here + + // accumulated counts must never exceed CODER_MAXSCALE + // as CODER_MAXSCALE is big enough, though, (2^29), this shouldn't happen and is not checked + + unsigned short* counts; + signed int local_symb; + unsigned int curr_total; + unsigned int curr_count; + unsigned int esc_prob; + int i; + + // make a local copy of the pointer + counts = context->counts; + + // check counts + if ( counts != NULL ) { // if counts are already set + // locally store current fill/symbol count + local_symb = sb0_count; + + // set the last symbol of the totals table_s zero + i = context->max_symbol - 1; + totals[ i + 2 ] = 0; + // (re)set current total + curr_total = 0; + + // go reverse though the whole counts table and accumulate counts + // leave space at the beginning of the table for the escape symbol + for ( ; i >= 0; i-- ) { + // only count probability if the current symbol is not 'scoreboard - excluded' + if ( scoreboard[ i ] == 0 ) { + curr_count = counts[ i ]; + if ( curr_count > 0 ) { + // add counts for the current symbol + curr_total = curr_total + curr_count; + // exclude symbol from scoreboard + scoreboard[ i ] = 1; + sb0_count--; + } + } + totals[ i + 1 ] = curr_total; + } + // here the escape calculation needs to take place + if ( local_symb == sb0_count ) + esc_prob = 1; + else if ( sb0_count == 0 ) + esc_prob = 0; + else { + // esc_prob = 1; + esc_prob = sb0_count * ( local_symb - sb0_count ); + esc_prob /= ( local_symb * context->max_count ); + esc_prob++; + } + // include escape probability in totals table + totals[ 0 ] = totals[ 1 ] + esc_prob; + } + else { // if counts are not already set + // setup counts for current table + context->counts = ( unsigned short* ) calloc( max_symbol, sizeof( short ) ); + if ( context->counts == NULL ) ERROR_EXIT; + // set totals table -> only escape probability included + totals[ 0 ] = 1; + totals[ 1 ] = 0; + } +} + + +/* ----------------------------------------------- + resizes one table by bitshifting each count using a specific value + ----------------------------------------------- */ + +inline void model_s::rescale_table( table_s* context, int scale_factor ) +{ + unsigned short* counts = context->counts; + int lst_symbol = context->max_symbol; + int i; + + // return now if counts not set + if ( counts == NULL ) return; + + // now scale the table by bitshifting each count + for ( i = 0; i < lst_symbol; i++ ) { + if ( counts[ i ] > 0 ) + counts[ i ] >>= scale_factor; + } + + // also rescale tables max count + context->max_count >>= scale_factor; + + // seek for new last symbol + for ( i = lst_symbol - 1; i >= 0; i-- ) + if ( counts[ i ] > 0 ) break; + context->max_symbol = i + 1; +} + + +/* ----------------------------------------------- + a recursive function to go through each context and rescale the counts + ----------------------------------------------- */ + +inline void model_s::recursive_flush( table_s* context, int scale_factor ) +{ + int i; + + // go through each link != NULL + if ( context->links != NULL ) + for ( i = 0; i < max_context; i++ ) + if ( context->links[ i ] != NULL ) + recursive_flush( context->links[ i ], scale_factor ); + + // rescale specific table + rescale_table( context, scale_factor ); +} + + +/* ----------------------------------------------- + frees all memory for all contexts starting at a given table_s + ----------------------------------------------- */ + +inline void model_s::recursive_cleanup( table_s *context ) +{ + // be careful not to cut any link too early! + + int i; + + // go through each link != NULL + if ( context->links != NULL ) { + for ( i = 0; i < max_context; i++ ) + if ( context->links[ i ] != NULL ) + recursive_cleanup( context->links[ i ] ); + free ( context->links ); + } + + // clean up table + if ( context->counts != NULL ) free ( context->counts ); + free( context ); +} + + +/* ----------------------------------------------- + special version of model_s for binary coding + ----------------------------------------------- */ + +model_b::model_b( int max_c, int max_o, int c_lim ) +{ + // boundaries of this model: + // ... (maximum symbol) -> 2 (0 or 1 ) + // max_c (maximum context) -> 1 <= max_c <= 1024 (???) + // max_o (maximum order) -> -1 <= max_o <= 4 + + table* null_table; + table* start_table; + int i; + + + // set error false + error = false; + + // copy settings into model + max_context = max_c; + max_order = max_o; + max_count = c_lim; + + + // set up null table + null_table = ( table* ) calloc( 1, sizeof( table ) ); + if ( null_table == NULL ) ERROR_EXIT; + + null_table->counts = ( unsigned short* ) calloc( 2, sizeof( short ) ); + if ( null_table->counts == NULL ) ERROR_EXIT; + null_table->counts[ 0 ] = 1; + null_table->counts[ 1 ] = 1; + null_table->scale = 2; + + // set up start table + start_table = ( table* ) calloc( 1, sizeof( table ) ); + if ( start_table == NULL ) ERROR_EXIT; + start_table->links = ( table** ) calloc( max_context, sizeof( table* ) ); + if ( start_table->links == NULL ) ERROR_EXIT; + start_table->scale = 0; + + // build links for start table & null table + start_table->lesser = null_table; + null_table->links = ( table** ) calloc( max_context, sizeof( table* ) ); + if ( null_table->links == NULL ) ERROR_EXIT; + for ( i = 0; i < max_context; i++ ) + null_table->links[ i ] = start_table; + + // alloc memory for storage & contexts + storage = ( table** ) calloc( max_order + 3, sizeof( table* ) ); + if ( storage == NULL ) ERROR_EXIT; + contexts = storage + 1; + + // integrate tables into contexts + contexts[ -1 ] = null_table; + contexts[ 0 ] = start_table; + + // build initial 'normal' tables + for ( i = 1; i <= max_order; i++ ) { + // set up current order table + contexts[ i ] = ( table* ) calloc( 1, sizeof( table ) ); + if ( contexts[ i ] == NULL ) ERROR_EXIT; + contexts[ i ]->scale = 0; + // build forward and backward links + contexts[ i ]->lesser = contexts[ i - 1 ]; + if ( i < max_order ) { + contexts[ i ]->links = ( table** ) calloc( max_context, sizeof( table* ) ); + if ( contexts[ i ]->links == NULL ) ERROR_EXIT; + } + else { + contexts[ i ]->links = NULL; + } + contexts[ i - 1 ]->links[ 0 ] = contexts[ i ]; + } +} + + +/* ----------------------------------------------- + model class destructor - recursive cleanup of memory is done here + ----------------------------------------------- */ + +model_b::~model_b( void ) +{ + table* context; + + + // clean up each 'normal' table + context = contexts[ 0 ]; + recursive_cleanup ( context ); + + // clean up null table + context = contexts[ -1 ]; + if ( context->links != NULL ) + free( context->links ); + if ( context->counts != NULL ) free( context->counts ); + free ( context ); + + // free everything else + free( storage ); +} + + +/* ----------------------------------------------- + updates statistics for a specific symbol / resets to highest order + ----------------------------------------------- */ + +void model_b::update_model( int symbol ) +{ + // use -1 if you just want to reset without updating statistics + + table* context = contexts[ max_order ]; + + // only contexts, that were actually used to encode + // the symbol get their counts updated + if ( ( symbol >= 0 ) && ( max_order >= 0 ) ) { + // update count for specific symbol & scale + context->counts[ symbol ]++; + context->scale++; + // if counts for that symbol have gone above the maximum count + // the table has to be resized (scale factor 2) + if ( context->counts[ symbol ] >= max_count ) + rescale_table( context, 1 ); + } +} + + +/* ----------------------------------------------- + shift in one context (max no of contexts is max_c) + ----------------------------------------------- */ + +void model_b::shift_context( int c ) +{ + table* context; + int i; + + // shifting is not possible if max_order is below 1 + // or context index is negative + if ( ( max_order < 1 ) || ( c < 0 ) ) return; + + // shift each orders' context + for ( i = max_order; i > 0; i-- ) { + // this is the new current order context + context = contexts[ i - 1 ]->links[ c ]; + + // check if context exists, build if needed + if ( context == NULL ) { + // reserve memory for next table + context = ( table* ) calloc( 1, sizeof( table ) ); + if ( context == NULL ) ERROR_EXIT; + // set internal counts NULL + context->counts = NULL; + context->scale = 0; + // link lesser context later if not existing, this is done below + context->lesser = contexts[ i - 2 ]->links[ c ]; + // finished here if this is a max order context + if ( i == max_order ) { + context->links = NULL; + } + else { + // build links to higher order tables otherwise + context->links = ( table** ) calloc( max_context, sizeof( table* ) ); + if ( context->links == NULL ) ERROR_EXIT; + // add lesser link for higher context (see above) + contexts[ i + 1 ]->lesser = context; + } + // put context to its right place + contexts[ i - 1 ]->links[ c ] = context; + } + + // switch context + contexts[ i ] = context; + } +} + + +/* ----------------------------------------------- + flushes the whole model by dividing through a specific scale factor + ----------------------------------------------- */ + +void model_b::flush_model( int scale_factor ) +{ + recursive_flush( contexts[ 0 ], scale_factor ); +} + + +/* ----------------------------------------------- + converts an int to a symbol, needed only when encoding + ----------------------------------------------- */ + +int model_b::convert_int_to_symbol( int c, symbol *s ) +{ + table* context = contexts[ max_order ]; + + // check if counts are available + check_counts( context ); + + // finding the scale is easy + s->scale = context->scale; + + // return high and low count for current symbol + if ( c == 0 ) { // if 0 is to be encoded + s->low_count = 0; + s->high_count = context->counts[ 0 ]; + } + else { // if 1 is to be encoded + s->low_count = context->counts[ 0 ]; + s->high_count = context->scale; + } + + return 1; +} + + +/* ----------------------------------------------- + returns the current context scale needed only when decoding + ----------------------------------------------- */ + +void model_b::get_symbol_scale( symbol *s ) +{ + table* context = contexts[ max_order ]; + + // check if counts are available + check_counts( context ); + + // getting the scale is easy + s->scale = context->scale; +} + + +/* ----------------------------------------------- + converts a count to an int, called after get_symbol_scale + ----------------------------------------------- */ + +int model_b::convert_symbol_to_int( int count, symbol *s ) +{ + table* context = contexts[ max_order ]; + unsigned short counts0 = context->counts[ 0 ]; + + // set up the current symbol + if ( count < counts0 ) { + s->low_count = 0; + s->high_count = counts0; + return 0; + } + else { + s->low_count = counts0; + s->high_count = s->scale; + return 1; + } +} + + +/* ----------------------------------------------- + this function checks if counts exist, and, if they exist and are below max + ----------------------------------------------- */ + +inline void model_b::check_counts( table *context ) +{ + unsigned short* counts = context->counts; + + // check if counts are available + if ( counts == NULL ) { + // setup counts for current table + counts = ( unsigned short* ) calloc( 2, sizeof( short ) ); + if ( counts == NULL ) ERROR_EXIT; + counts[ 0 ] = 1; + counts[ 1 ] = 1; + // set scale + context->counts = counts; + context->scale = 2; + } +} + + +/* ----------------------------------------------- + resizes one table by bitshifting each count using a specific value + ----------------------------------------------- */ + +inline void model_b::rescale_table( table* context, int scale_factor ) +{ + unsigned short* counts = context->counts; + + // return now if counts not set + if ( counts == NULL ) return; + + // now scale the table by bitshifting each count, be careful not to set any count zero + counts[ 0 ] >>= scale_factor; + counts[ 1 ] >>= scale_factor; + if ( counts[ 0 ] == 0 ) counts[ 0 ] = 1; + if ( counts[ 1 ] == 0 ) counts[ 1 ] = 1; + context->scale = counts[ 0 ] + counts[ 1 ]; +} + + +/* ----------------------------------------------- + a recursive function to go through each context and rescale the counts + ----------------------------------------------- */ + +inline void model_b::recursive_flush( table* context, int scale_factor ) +{ + int i; + + // go through each link != NULL + if ( context->links != NULL ) + for ( i = 0; i < max_context; i++ ) + if ( context->links[ i ] != NULL ) + recursive_flush( context->links[ i ], scale_factor ); + + // rescale specific table + rescale_table( context, scale_factor ); +} + + +/* ----------------------------------------------- + frees all memory for all contexts starting at a given table + ----------------------------------------------- */ + +inline void model_b::recursive_cleanup( table *context ) +{ + int i; + + // go through each link != NULL + if ( context->links != NULL ) { + for ( i = 0; i < max_context; i++ ) + if ( context->links[ i ] != NULL ) + recursive_cleanup( context->links[ i ] ); + free ( context->links ); + } + + // clean up table + if ( context->counts != NULL ) free ( context->counts ); + free( context ); +} diff --git a/contrib/preflate/packARI/source/aricoder.h b/contrib/preflate/packARI/source/aricoder.h new file mode 100644 index 0000000..df73286 --- /dev/null +++ b/contrib/preflate/packARI/source/aricoder.h @@ -0,0 +1,268 @@ +// defines for coder +#define CODER_USE_BITS 31 // must never be above 31 +#define CODER_LIMIT100 ( (unsigned int) ( 1 << CODER_USE_BITS ) ) +#define CODER_LIMIT025 ( ( CODER_LIMIT100 / 4 ) * 1 ) +#define CODER_LIMIT050 ( ( CODER_LIMIT100 / 4 ) * 2 ) +#define CODER_LIMIT075 ( ( CODER_LIMIT100 / 4 ) * 3 ) +#define CODER_MAXSCALE CODER_LIMIT025 - 1 +#define ESCAPE_SYMBOL CODER_LIMIT025 + + +// symbol struct, used in arithmetic coding +struct symbol { + unsigned int low_count; + unsigned int high_count; + unsigned int scale; +}; + +// table struct, used in in statistical models, +// holding all info needed for one context +struct table { + // counts for each symbol contained in the table + unsigned short* counts; + // links to higher order contexts + struct table** links; + // link to lower order context + struct table* lesser; + // accumulated counts + unsigned int scale; +}; + +// special table struct, used in in model_s, +// holding additional info for a speedier 'totalize_table' +struct table_s { + // counts for each symbol contained in the table + unsigned short* counts; + // links to higher order contexts + struct table_s** links; + // link to lower order context + struct table_s* lesser; + // speedup info + unsigned short max_count; + unsigned short max_symbol; + // unsigned short esc_prob; +}; + + +/* ----------------------------------------------- + class for arithmetic coding of data to/from iostream + ----------------------------------------------- */ + +class aricoder +{ + public: + aricoder( iostream* stream, int iomode ); + ~aricoder( void ); + void encode( symbol* s ); + unsigned int decode_count( symbol* s ); + void decode( symbol* s ); + + private: + // bitwise operations + void write_bit( unsigned char bit ); + unsigned char read_bit( void ); + + // i/o variables + iostream* sptr; + int mode; + unsigned char bbyte; + unsigned char cbit; + + // arithmetic coding variables + unsigned int ccode; + unsigned int clow; + unsigned int chigh; + unsigned int cstep; + unsigned int nrbits; +}; + + +/* ----------------------------------------------- + universal statistical model for arithmetic coding + ----------------------------------------------- */ + +class model_s +{ + public: + + model_s( int max_s, int max_c, int max_o, int c_lim ); + ~model_s( void ); + + void update_model( int symbol ); + void shift_context( int c ); + void flush_model( int scale_factor ); + void exclude_symbols( char rule, int c ); + + int convert_int_to_symbol( int c, symbol *s ); + void get_symbol_scale( symbol *s ); + int convert_symbol_to_int( int count, symbol *s ); + + bool error; + + + private: + + // unsigned short* totals; + unsigned int* totals; + char* scoreboard; + int sb0_count; + table_s **contexts; + table_s **storage; + + int max_symbol; + int max_context; + int current_order; + int max_order; + int max_count; + + inline void totalize_table(table_s* context ); + inline void rescale_table(table_s* context, int scale_factor ); + inline void recursive_flush(table_s* context, int scale_factor ); + inline void recursive_cleanup(table_s* context ); +}; + + +/* ----------------------------------------------- + binary statistical model for arithmetic coding + ----------------------------------------------- */ + +class model_b +{ + public: + + model_b( int max_c, int max_o, int c_lim ); + ~model_b( void ); + + void update_model( int symbol ); + void shift_context( int c ); + void flush_model( int scale_factor ); + + int convert_int_to_symbol( int c, symbol *s ); + void get_symbol_scale( symbol *s ); + int convert_symbol_to_int( int count, symbol *s ); + + bool error; + + + private: + + table **contexts; + table **storage; + + int max_context; + int max_order; + int max_count; + + inline void check_counts( table *context ); + inline void rescale_table( table* context, int scale_factor ); + inline void recursive_flush( table* context, int scale_factor ); + inline void recursive_cleanup( table *context ); +}; + + +/* ----------------------------------------------- + shift context x2 model_s function + ----------------------------------------------- */ +static inline void shift_model( model_s* model, int ctx1, int ctx2 ) +{ + model->shift_context( ctx1 ); + model->shift_context( ctx2 ); +} + + +/* ----------------------------------------------- + shift context x3 model_s function + ----------------------------------------------- */ +static inline void shift_model( model_s* model, int ctx1, int ctx2, int ctx3 ) +{ + model->shift_context( ctx1 ); + model->shift_context( ctx2 ); + model->shift_context( ctx3 ); +} + + +/* ----------------------------------------------- + shift context x2 model_b function + ----------------------------------------------- */ +static inline void shift_model( model_b* model, int ctx1, int ctx2 ) +{ + model->shift_context( ctx1 ); + model->shift_context( ctx2 ); +} + + +/* ----------------------------------------------- + shift context x3 model_b function + ----------------------------------------------- */ +static inline void shift_model( model_b* model, int ctx1, int ctx2, int ctx3 ) +{ + model->shift_context( ctx1 ); + model->shift_context( ctx2 ); + model->shift_context( ctx3 ); +} + + +/* ----------------------------------------------- + generic model_s encoder function + ----------------------------------------------- */ +static inline void encode_ari( aricoder* encoder, model_s* model, int c ) +{ + static symbol s; + static int esc; + + do { + esc = model->convert_int_to_symbol( c, &s ); + encoder->encode( &s ); + } while ( esc ); + model->update_model( c ); +} + +/* ----------------------------------------------- + generic model_s decoder function + ----------------------------------------------- */ +static inline int decode_ari( aricoder* decoder, model_s* model ) +{ + static symbol s; + static unsigned int count; + static int c; + + do{ + model->get_symbol_scale( &s ); + count = decoder->decode_count( &s ); + c = model->convert_symbol_to_int( count, &s ); + decoder->decode( &s ); + } while ( c == ESCAPE_SYMBOL ); + model->update_model( c ); + + return c; +} + +/* ----------------------------------------------- + generic model_b encoder function + ----------------------------------------------- */ +static inline void encode_ari( aricoder* encoder, model_b* model, int c ) +{ + static symbol s; + + model->convert_int_to_symbol( c, &s ); + encoder->encode( &s ); + model->update_model( c ); +} + +/* ----------------------------------------------- + generic model_b decoder function + ----------------------------------------------- */ +static inline int decode_ari( aricoder* decoder, model_b* model ) +{ + static symbol s; + static unsigned int count; + static int c; + + model->get_symbol_scale( &s ); + count = decoder->decode_count( &s ); + c = model->convert_symbol_to_int( count, &s ); + decoder->decode( &s ); + model->update_model( c ); + + return c; +} diff --git a/contrib/preflate/packARI/source/bitops.cpp b/contrib/preflate/packARI/source/bitops.cpp new file mode 100644 index 0000000..ba80de9 --- /dev/null +++ b/contrib/preflate/packARI/source/bitops.cpp @@ -0,0 +1,926 @@ +/* +This file contains special classes for bitwise +reading and writing of arrays +*/ + +#include +#include +#include +#include "bitops.h" + +#ifdef _MSC_VER // Changed by schnaader to enable compilation by MSVC +#include +#define setmode _setmode +#endif + +#define BUFFER_SIZE 1024 * 1024 + + +// special realloc with guaranteed free() of previous memory +static inline void* frealloc( void* ptr, size_t size ) { + void* n_ptr = realloc( ptr, (size) ? size : 1 ); + if ( n_ptr == NULL ) free( ptr ); + return n_ptr; +} + + +/* ----------------------------------------------- + constructor for abitreader class + ----------------------------------------------- */ + +abitreader::abitreader( unsigned char* array, int size ) +{ + cbyte = 0; + cbit = 8; + peof = 0; + eof = false; + + data = array; + lbyte = size; +} + +/* ----------------------------------------------- + destructor for abitreader class + ----------------------------------------------- */ + +abitreader::~abitreader( void ) +{ +} + +/* ----------------------------------------------- + reads n bits from abitreader + ----------------------------------------------- */ + +unsigned int abitreader::read( int nbits ) +{ + unsigned int retval = 0; + + // safety check for eof + if ( eof ) { + peof += nbits; + return 0; + } + + while ( nbits >= cbit ) { + nbits -= cbit; + retval |= ( RBITS( data[cbyte], cbit ) << nbits ); + cbit = 8; + if ( ++cbyte >= lbyte ) { + peof = nbits; + eof = true; + return retval; + } + } + + if ( nbits > 0 ) { + retval |= ( MBITS( data[cbyte], cbit, (cbit-nbits) ) ); + cbit -= nbits; + } + + return retval; +} + +/* ----------------------------------------------- + reads one bit from abitreader + ----------------------------------------------- */ + +unsigned char abitreader::read_bit( void ) +{ + unsigned char bit; + + // safety check for eof + if (eof) { + peof++; + return 0; + } + + // read one bit + bit = BITN( data[cbyte], --cbit ); + if ( cbit == 0 ) { + if ( ++cbyte == lbyte ) eof = true; + cbit = 8; + } + + return bit; +} + +/* ----------------------------------------------- + to skip padding from current byte + ----------------------------------------------- */ + +unsigned char abitreader::unpad( unsigned char fillbit ) +{ + if ( ( cbit == 8 ) || eof ) return fillbit; + else { + fillbit = read( 1 ); + while ( cbit != 8 ) read( 1 ); + } + + return fillbit; +} + +/* ----------------------------------------------- + get current position in array + ----------------------------------------------- */ + +int abitreader::getpos( void ) +{ + return cbyte; +} + +/* ----------------------------------------------- + get current bit position + ----------------------------------------------- */ + +int abitreader::getbitp( void ) +{ + return cbit; +} + +/* ----------------------------------------------- + set byte and bit position + ----------------------------------------------- */ + +void abitreader::setpos( int pbyte, int pbit ) +{ + if ( pbyte < lbyte ) { + // reset eof + eof = false; + // set positions + cbyte = pbyte; + cbit = pbit; + } else { + // set eof + eof = true; + // set positions + cbyte = lbyte; + cbit = 8; + peof = ( ( pbyte - lbyte ) * 8 ) + 8 - pbit; + } +} + +/* ----------------------------------------------- + rewind n bits + ----------------------------------------------- */ + +void abitreader::rewind_bits( int nbits ) +{ + if ( eof ) { + if ( nbits > peof ) nbits -= peof; + else { + peof -= nbits; + return; + } + eof = false; + } + for ( cbit += nbits; cbit > 8; cbyte--, cbit -= 8 ); + if ( cbyte < 0 ) { + cbyte = 0; + cbit = 8; + } +} + + +/* ----------------------------------------------- + constructor for abitwriter class + ----------------------------------------------- */ + +abitwriter::abitwriter( int size ) +{ + fillbit = 1; + adds = 65536; + cbyte = 0; + cbit = 8; + + error = false; + fmem = true; + + dsize = ( size > 0 ) ? size : adds; + data = ( unsigned char* ) malloc ( dsize ); + if ( data == NULL ) { + error = true; + return; + } + + // fill buffer with zeroes + memset( data, 0, dsize * sizeof( char ) ); + // for ( int i = 0; i < dsize; i++ ) data[i] = 0; +} + +/* ----------------------------------------------- + destructor for abitwriter class + ----------------------------------------------- */ + +abitwriter::~abitwriter( void ) +{ + // free memory if pointer was not given out + if ( fmem ) free( data ); +} + +/* ----------------------------------------------- + writes n bits to abitwriter + ----------------------------------------------- */ + +void abitwriter::write( unsigned int val, int nbits ) +{ + // safety check for error + if ( error ) return; + + // test if pointer beyond flush treshold + if ( cbyte > ( dsize - 5 ) ) { + dsize += adds; + data = (unsigned char*) frealloc( data, dsize ); + if ( data == NULL ) { + error = true; + return; + } + memset( ( data + cbyte + 1 ), 0, ( dsize - ( cbyte + 1 ) ) * sizeof( char ) ); + } + + // write data + while ( nbits >= cbit ) { + data[cbyte] |= ( MBITS32(val, nbits, (nbits-cbit)) ); + nbits -= cbit; + cbyte++; + cbit = 8; + } + + if ( nbits > 0 ) { + data[cbyte] |= ( (RBITS32(val, nbits)) << (cbit - nbits) ); + cbit -= nbits; + } +} + +/* ----------------------------------------------- + writes one bit to abitwriter + ----------------------------------------------- */ + +void abitwriter::write_bit( unsigned char bit ) +{ + // safety check for error + if ( error ) return; + + // write data + if ( bit ) data[cbyte] |= 0x1 << (--cbit); + else --cbit; + if ( cbit == 0 ) { + // test if pointer beyond flush treshold + if ( ++cbyte > ( dsize - 5 ) ) { + dsize += adds; + data = (unsigned char*) frealloc( data, dsize ); + if ( data == NULL ) { + error = true; + return; + } + memset( ( data + cbyte + 1 ), 0, ( dsize - ( cbyte + 1 ) ) * sizeof( char ) ); + } + cbit = 8; + } +} + +/* ----------------------------------------------- + pads data using fillbit + ----------------------------------------------- */ + +void abitwriter::pad( unsigned char fillbit ) +{ + while ( cbit < 8 ) + write( fillbit, 1 ); +} + +/* ----------------------------------------------- + gets data array from abitwriter + ----------------------------------------------- */ + +unsigned char* abitwriter::getptr( void ) +{ + // data is padded here + pad( fillbit ); + // forbid freeing memory + fmem = false; + // realloc data + data = (unsigned char*) frealloc( data, cbyte ); + + return data; +} + +/* ----------------------------------------------- + gets size of data array from abitwriter + ----------------------------------------------- */ + +int abitwriter::getpos( void ) +{ + return cbyte; +} + +/* ----------------------------------------------- + get current bit position + ----------------------------------------------- */ + +int abitwriter::getbitp( void ) +{ + return cbit; +} + + +/* ----------------------------------------------- + constructor for abytewriter class + ----------------------------------------------- */ + +abytereader::abytereader( unsigned char* array, int size ) +{ + cbyte = 0; + eof = false; + + data = array; + lbyte = size; + + if ( ( data == NULL ) || ( lbyte == 0 ) ) + eof = true; +} + +/* ----------------------------------------------- + destructor for abytewriter class + ----------------------------------------------- */ + +abytereader::~abytereader( void ) +{ +} + +/* ----------------------------------------------- + reads 1 byte from abytereader + ----------------------------------------------- */ + +int abytereader::read( unsigned char* byte ) +{ + if ( cbyte >= lbyte ) { + cbyte = lbyte; + eof = true; + return 0; + } + else { + *byte = data[ cbyte++ ]; + return 1; + } +} + +/* ----------------------------------------------- + reads n bytes from abytereader + ----------------------------------------------- */ + +int abytereader::read_n( unsigned char* byte, int n ) +{ + int nl = lbyte - cbyte; + int i; + + if ( nl < n ) { + for ( i = 0; i < nl; i++ ) + byte[ i ] = data[ cbyte + i ]; + cbyte = lbyte; + eof = true; + return nl; + } + else { + for ( i = 0; i < n; i++ ) + byte[ i ] = data[ cbyte + i ]; + cbyte += n; + return n; + } +} + +/* ----------------------------------------------- + go to position in data + ----------------------------------------------- */ + +void abytereader::seek( int pos ) +{ + if ( pos >= lbyte ) { + cbyte = lbyte; + eof = true; + } + else { + cbyte = pos; + eof = false; + } +} + +/* ----------------------------------------------- + gets size of current data + ----------------------------------------------- */ + +int abytereader::getsize( void ) +{ + return lbyte; +} + +/* ----------------------------------------------- + gets current position from abytereader + ----------------------------------------------- */ + +int abytereader::getpos( void ) +{ + return cbyte; +} + + +/* ----------------------------------------------- + constructor for abytewriter class + ----------------------------------------------- */ + +abytewriter::abytewriter( int size ) +{ + adds = 65536; + cbyte = 0; + + error = false; + fmem = true; + + dsize = ( size > 0 ) ? size : adds; + data = (unsigned char*) malloc( dsize ); + if ( data == NULL ) { + error = true; + return; + } +} + +/* ----------------------------------------------- + destructor for abytewriter class + ----------------------------------------------- */ + +abytewriter::~abytewriter( void ) +{ + // free data if pointer is not read + if ( fmem ) free( data ); +} + +/* ----------------------------------------------- + writes 1 byte to abytewriter + ----------------------------------------------- */ + +void abytewriter::write( unsigned char byte ) +{ + // safety check for error + if ( error ) return; + + // test if pointer beyond flush threshold + if ( cbyte >= ( dsize - 2 ) ) { + dsize += adds; + data = (unsigned char*) frealloc( data, dsize ); + if ( data == NULL ) { + error = true; + return; + } + } + + // write data + data[ cbyte++ ] = byte; +} + +/* ----------------------------------------------- + writes n byte to abytewriter + ----------------------------------------------- */ + +void abytewriter::write_n( unsigned char* byte, int n ) +{ + // safety check for error + if ( error ) return; + + // make sure that pointer doesn't get beyond flush threshold + while ( ( cbyte + n ) >= ( dsize - 2 ) ) { + dsize += adds; + data = (unsigned char*) frealloc( data, dsize ); + if ( data == NULL ) { + error = true; + return; + } + } + + // copy data from array + while ( n-- > 0 ) + data[ cbyte++ ] = *(byte++); +} + +/* ----------------------------------------------- + gets data array from abytewriter + ----------------------------------------------- */ + +unsigned char* abytewriter::getptr( void ) +{ + // safety check for error + if ( error ) return NULL; + // forbid freeing memory + fmem = false; + // realloc data + data = (unsigned char*) frealloc( data, cbyte ); + + return data; +} + +/* ----------------------------------------------- + peeks into data array from abytewriter + ----------------------------------------------- */ + +unsigned char* abytewriter::peekptr( void ) +{ + return data; +} + +/* ----------------------------------------------- + gets size of data array from abytewriter + ----------------------------------------------- */ + +int abytewriter::getpos( void ) +{ + return cbyte; +} + +/* ----------------------------------------------- + reset without realloc + ----------------------------------------------- */ + +void abytewriter::reset( void ) +{ + // set position of current byte + cbyte = 0; +} + + +/* ----------------------------------------------- + constructor for iostream class + ----------------------------------------------- */ + +iostream::iostream( void* src, int srctype, int srcsize, int iomode ) +{ + // locally copy source, source type # and io mode # + source = src; + srct = srctype; + srcs = srcsize; + mode = iomode; + + // don't free memory when reading - this will be useful if switching occurs + free_mem_sw = false; + + // set binary mode for streams + #if defined( _WIN32 ) + setmode( fileno( stdin ), O_BINARY ); + setmode( fileno( stdout ), O_BINARY ); + #endif + + // open file/mem/stream + switch ( srct ) + { + case 0: + open_file(); + break; + + case 1: + open_mem(); + break; + + case 2: + open_stream(); + break; + + default: + break; + } +} + +/* ----------------------------------------------- + destructor for iostream class + ----------------------------------------------- */ + +iostream::~iostream( void ) +{ + // if needed, write memory to stream or free memory from buffered stream + if ( srct == 2 ) { + if ( mode == 1 ) { + if ( !(mwrt->error) ) { + srcs = mwrt->getpos(); + source = mwrt->getptr(); + fwrite( source, sizeof( char ), srcs, stdout ); + } + } + } + + // free all buffers + if ( srct == 0 ) { + if ( fptr != NULL ) { + if ( mode == 1 ) fflush( fptr ); + fclose( fptr ); + } + } + else if ( mode == 0 ) { + if ( free_mem_sw ) + free( source ); + delete( mrdr ); + } + else + delete( mwrt ); +} + +/* ----------------------------------------------- + switches mode from reading to writing and vice versa + ----------------------------------------------- */ + +void iostream::switch_mode( void ) +{ + // return immediately if there's an error + if ( chkerr() ) return; + + + if ( mode == 0 ) { + // WARNING: when switching from reading to writing, information might be lost forever + switch ( srct ) { + case 0: + fclose( fptr ); + fptr = fopen( ( char* ) source, "wb" ); + break; + case 1: + case 2: + delete( mrdr ); + if ( free_mem_sw ) + free( source ); // see? I've told you so :-) + mwrt = new abytewriter( srcs ); + break; + default: + break; + } + mode = 1; + } + else { + // switching from writing to reading is a bit more complicated + switch ( srct ) { + case 0: + fflush( fptr ); + fclose( fptr ); + fptr = fopen( ( char* ) source, "rb" ); + break; + case 1: + case 2: + source = mwrt->getptr(); + srcs = mwrt->getpos(); + delete( mwrt ); + mrdr = new abytereader( ( unsigned char* ) source, srcs ); + free_mem_sw = true; + break; + default: + break; + } + mode = 0; + } +} + +/* ----------------------------------------------- + generic read function + ----------------------------------------------- */ + +int iostream::read( void* to, int tpsize, int dtsize ) +{ + return ( srct == 0 ) ? read_file( to, tpsize, dtsize ) : read_mem( to, tpsize, dtsize ); +} + +/* ----------------------------------------------- + generic write function + ----------------------------------------------- */ + +int iostream::write( void* from, int tpsize, int dtsize ) +{ + return ( srct == 0 ) ? write_file( from, tpsize, dtsize ) : write_mem( from, tpsize, dtsize ); +} + +/* ----------------------------------------------- + flush function + ----------------------------------------------- */ + +int iostream::flush( void ) +{ + if ( srct == 0 ) + fflush( fptr ); + + return getpos(); +} + +/* ----------------------------------------------- + rewind to beginning of stream + ----------------------------------------------- */ + +int iostream::rewind( void ) +{ + // WARNING: when writing, rewind might lose all your data + if ( srct == 0 ) + fseek( fptr, 0, SEEK_SET ); + else if ( mode == 0 ) + mrdr->seek( 0 ); + else + mwrt->reset(); + + return getpos(); +} + +/* ----------------------------------------------- + get current position in stream + ----------------------------------------------- */ + +int iostream::getpos( void ) +{ + int pos; + + if ( srct == 0 ) + pos = ftell( fptr ); + else if ( mode == 0 ) + pos = mrdr->getpos(); + else + pos = mwrt->getpos(); + + return pos; +} + +/* ----------------------------------------------- + get size of file + ----------------------------------------------- */ + +int iostream::getsize( void ) +{ + int pos; + int siz; + + if ( mode == 0 ) { + if ( srct == 0 ) { + pos = ftell( fptr ); + fseek( fptr, 0, SEEK_END ); + siz = ftell( fptr ); + fseek( fptr, pos, SEEK_SET ); + } + else { + siz = mrdr->getsize(); + } + } + else { + siz = getpos(); + } + + return siz; +} + +/* ----------------------------------------------- + get data pointer (for mem io only) + ----------------------------------------------- */ + +unsigned char* iostream::getptr( void ) +{ + if ( srct == 1 ) + return ( mode == 0 ) ? ( unsigned char* ) source : mwrt->getptr(); + else + return NULL; +} + +/* ----------------------------------------------- + check for errors + ----------------------------------------------- */ + +bool iostream::chkerr( void ) +{ + bool error = false; + + // check for user input errors + if ( ( mode != 0 ) && ( mode != 1 ) ) + error = true; + if ( ( srct != 0 ) && ( srct != 1 ) && ( srct != 2 ) ) + error = true; + + // check for io errors + if ( srct == 0 ) { + if ( fptr == NULL ) + error = true; + else if ( ferror( fptr ) ) + error = true; + } + else if ( mode == 0 ) { + if ( mrdr == NULL ) + error = true; + } + else { + if ( mwrt == NULL ) + error = true; + else if ( mwrt->error ) + error = true; + } + + return error; +} + +/* ----------------------------------------------- + check for eof (read only) + ----------------------------------------------- */ + +bool iostream::chkeof( void ) +{ + if ( mode == 0 ) + return ( srct == 0 ) ? feof( fptr ) : mrdr->eof; + else + return false; +} + +/* ----------------------------------------------- + open function for files + ----------------------------------------------- */ + +void iostream::open_file( void ) +{ + char* fn = (char*) source; + + // open file for reading / writing + fptr = fopen( fn, ( mode == 0 ) ? "rb" : "wb" ); +} + +/* ----------------------------------------------- + open function for memory + ----------------------------------------------- */ + +void iostream::open_mem( void ) +{ + if ( mode == 0 ) + mrdr = new abytereader( ( unsigned char* ) source, srcs ); + else + mwrt = new abytewriter( srcs ); +} + +/* ----------------------------------------------- + open function for streams + ----------------------------------------------- */ + +void iostream::open_stream( void ) +{ + abytewriter* strwrt; + unsigned char* buffer; + int i; + + if ( mode == 0 ) { + // read whole stream into memory buffer + strwrt = new abytewriter( 0 ); + buffer = ( unsigned char* ) calloc( BUFFER_SIZE, sizeof( char ) ); + if ( buffer != NULL ) { + while ( ( i = fread( buffer, sizeof( char ), BUFFER_SIZE, stdin ) ) > 0 ) + strwrt->write_n( buffer, i ); + } + if ( strwrt->error ) { + source = NULL; + srcs = 0; + } + else { + source = strwrt->getptr(); + srcs = strwrt->getpos(); + } + delete ( strwrt ); + free( buffer ); + // free memory after done + free_mem_sw = true; + } + + // for writing: simply open new stream in mem writer + // writing to stream will be done later + open_mem(); +} + +/* ----------------------------------------------- + write function for files + ----------------------------------------------- */ + +int iostream::write_file( void* from, int tpsize, int dtsize ) +{ + return fwrite( from, tpsize, dtsize, fptr ); +} + +/* ----------------------------------------------- + read function for files + ----------------------------------------------- */ + +int iostream::read_file( void* to, int tpsize, int dtsize ) +{ + return fread( to, tpsize, dtsize, fptr ); +} + +/* ----------------------------------------------- + write function for memory + ----------------------------------------------- */ + +int iostream::write_mem( void* from, int tpsize, int dtsize ) +{ + int n = tpsize * dtsize; + + mwrt->write_n( ( unsigned char* ) from, n ); + + return ( mwrt->error ) ? 0 : n; +} + +/* ----------------------------------------------- + read function for memory + ----------------------------------------------- */ + +int iostream::read_mem( void* to, int tpsize, int dtsize ) +{ + int n = tpsize * dtsize; + + return ( mrdr->read_n( ( unsigned char* ) to, n ) ) / tpsize; +} diff --git a/contrib/preflate/packARI/source/bitops.h b/contrib/preflate/packARI/source/bitops.h new file mode 100644 index 0000000..db12311 --- /dev/null +++ b/contrib/preflate/packARI/source/bitops.h @@ -0,0 +1,168 @@ +#define RBITS( c, n ) ( c & ( 0xFF >> (8 - n) ) ) +#define LBITS( c, n ) ( c >> (8 - n) ) +#define MBITS( c, l, r ) ( RBITS( c,l ) >> r ) +#define RBITS16( c, n ) ( c & ( 0xFFFFFFFF >> (16 - n) ) ) +#define LBITS16( c, n ) ( c >> (16 - n) ) +#define MBITS16( c, l, r ) ( RBITS16( c,l ) >> r ) +#define RBITS32( c, n ) ( c & ( 0xFFFFFFFF >> (32 - n) ) ) +#define LBITS32( c, n ) ( c >> (32 - n) ) +#define MBITS32( c, l, r ) ( RBITS32( c,l ) >> r ) +#define BITN( c, n ) ( (c >> n) & 0x1 ) +#define BITLEN( l, v ) for ( l = 0; ( v >> l ) > 0; l++ ) +#define FDIV2( v, p ) ( ( v < 0 ) ? -( (-v) >> p ) : ( v >> p ) ) + +#define TYPE_FILE 0 +#define TYPE_MEMORY 1 +#define TYPE_STREAM 2 +#define MODE_READ 0 +#define MODE_WRITE 1 + +#define BTST_BUFF 1024 * 1024 + +#include + + +/* ----------------------------------------------- + class to read arrays bitwise + ----------------------------------------------- */ + +class abitreader +{ +public: + abitreader( unsigned char* array, int size ); + ~abitreader( void ); + unsigned int read( int nbits ); + unsigned char read_bit( void ); + unsigned char unpad( unsigned char fillbit ); + int getpos( void ); + int getbitp( void ); + void setpos( int pbyte, int pbit ); + void rewind_bits( int nbits ); + bool eof; + int peof; + +private: + unsigned char* data; + int lbyte; + int cbyte; + int cbit; +}; + + +/* ----------------------------------------------- + class to write arrays bitwise + ----------------------------------------------- */ + +class abitwriter +{ +public: + abitwriter( int size ); + ~abitwriter( void ); + void write( unsigned int val, int nbits ); + void write_bit( unsigned char bit ); + void pad ( unsigned char fillbit ); + unsigned char* getptr( void ); + int getpos( void ); + int getbitp( void ); + bool error; + unsigned char fillbit; + +private: + unsigned char* data; + int dsize; + int adds; + int cbyte; + int cbit; + bool fmem; +}; + + +/* ----------------------------------------------- + class to read arrays bytewise + ----------------------------------------------- */ + +class abytereader +{ +public: + abytereader( unsigned char* array, int size ); + ~abytereader( void ); + int read( unsigned char* byte ); + int read_n( unsigned char* byte, int n ); + void seek( int pos ); + int getsize( void ); + int getpos( void ); + bool eof; + +private: + unsigned char* data; + int lbyte; + int cbyte; +}; + + +/* ----------------------------------------------- + class to write arrays bytewise + ----------------------------------------------- */ + +class abytewriter +{ +public: + abytewriter( int size ); + ~abytewriter( void ); + void write( unsigned char byte ); + void write_n( unsigned char* byte, int n ); + unsigned char* getptr( void ); + unsigned char* peekptr( void ); + int getpos( void ); + void reset( void ); + bool error; + +private: + unsigned char* data; + int dsize; + int adds; + int cbyte; + bool fmem; +}; + + +/* ----------------------------------------------- + class for input and output from file or memory + ----------------------------------------------- */ + +class iostream +{ +public: + iostream( void* src, int srctype, int srcsize, int iomode ); + ~iostream( void ); + void switch_mode( void ); + int read( void* to, int tpsize, int dtsize ); + int write( void* from, int tpsize, int dtsize ); + int flush( void ); + int rewind( void ); + int getpos( void ); + int getsize( void ); + unsigned char* getptr( void ); + bool chkerr( void ); + bool chkeof( void ); + +private: + void open_file( void ); + void open_mem( void ); + void open_stream( void ); + + int write_file( void* from, int tpsize, int dtsize ); + int read_file( void* to, int tpsize, int dtsize ); + int write_mem( void* from, int tpsize, int dtsize ); + int read_mem( void* to, int tpsize, int dtsize ); + + FILE* fptr; + abytewriter* mwrt; + abytereader* mrdr; + + bool free_mem_sw; + void* source; + int mode; + int srct; + int srcs; +}; diff --git a/contrib/preflate/packARI/source/file_icon.ico b/contrib/preflate/packARI/source/file_icon.ico new file mode 100644 index 0000000..16fbf62 Binary files /dev/null and b/contrib/preflate/packARI/source/file_icon.ico differ diff --git a/contrib/preflate/packARI/source/icons.res b/contrib/preflate/packARI/source/icons.res new file mode 100644 index 0000000..67d7217 Binary files /dev/null and b/contrib/preflate/packARI/source/icons.res differ diff --git a/contrib/preflate/packARI/source/packari.cpp b/contrib/preflate/packARI/source/packari.cpp new file mode 100644 index 0000000..dce6698 --- /dev/null +++ b/contrib/preflate/packARI/source/packari.cpp @@ -0,0 +1,1751 @@ +#include +#include +#include +#include + +#include "bitops.h" +#include "aricoder.h" +#include "paritbl.h" + +#if defined BUILD_DLL // define BUILD_LIB from the compiler options if you want to compile a DLL! + #define BUILD_LIB +#endif + +#if defined BUILD_LIB // define BUILD_LIB from the compiler options if you want to compile a library! + #include "packarilib.h" +#endif + +#define INTERN static + +#define INIT_MODEL_S(a,b,c) new model_s( a, b, c, 511 ) +#define INIT_MODEL_B(a,b) new model_b( a, b, 511 ) + +#define ABS(v1) ( (v1 < 0) ? -v1 : v1 ) +#define ABSDIFF(v1,v2) ( (v1 > v2) ? (v1 - v2) : (v2 - v1) ) +#define ROUND_F(v1) ( (v1 < 0) ? (int) (v1 - 0.5) : (int) (v1 + 0.5) ) +#define CLAMPED(l,h,v) ( ( v < l ) ? l : ( v > h ) ? h : v ) + +#define MEM_ERRMSG "out of memory error" +#define FRD_ERRMSG "could not read file / file not found" +#define FWR_ERRMSG "could not write file / file write-protected" +#define MSG_SIZE 128 +#define BARLEN 36 + + + +/* ----------------------------------------------- + function declarations: main interface + ----------------------------------------------- */ + +#if !defined( BUILD_LIB ) +INTERN void initialize_options( int argc, char** argv ); +INTERN void process_ui( void ); +INTERN inline const char* get_status( bool (*function)() ); +INTERN void show_help( void ); +#endif +INTERN void process_file( void ); +INTERN void execute( bool (*function)() ); + + +/* ----------------------------------------------- + function declarations: main functions + ----------------------------------------------- */ + +#if !defined( BUILD_LIB ) +INTERN bool check_file( void ); +INTERN bool swap_streams( void ); +INTERN bool compare_output( void ); +#endif +INTERN bool reset_buffers( void ); +INTERN bool decide_model_order( void ); +INTERN bool encode_file( void ); +INTERN bool decode_file( void ); + + +/* ----------------------------------------------- + function declarations: other coding functions + ----------------------------------------------- */ + +INTERN inline bool check_par_header( void ); +INTERN inline bool read_par_header( void ); +INTERN inline bool write_par_header( void ); +INTERN inline bool endecode_file( bool encoding ); +INTERN inline double cost_count_ari( model_s* model, int c ); + + +/* ----------------------------------------------- + function declarations: miscelaneous helpers + ----------------------------------------------- */ + +#if !defined( BUILD_LIB ) +INTERN inline void progress_bar( int current, int last ); +INTERN inline char* create_filename( const char* base, const char* extension ); +INTERN inline char* unique_filename( const char* base, const char* extension ); +INTERN inline void set_extension( const char* filename, const char* extension ); +INTERN inline void add_underscore( char* filename ); +#endif +INTERN inline bool file_exists( const char* filename ); + + +/* ----------------------------------------------- + function declarations: developers functions + ----------------------------------------------- */ + +// these are developers functions, they are not needed +// in any way to compress or decompress files +#if !defined(BUILD_LIB) && defined(DEV_BUILD) +INTERN bool write_errfile( void ); +#endif + +/* ----------------------------------------------- + global variables: library only variables + ----------------------------------------------- */ +#if defined(BUILD_LIB) +INTERN int lib_in_type = -1; +INTERN int lib_out_type = -1; +#endif + + +/* ----------------------------------------------- + global variables: info about files + ----------------------------------------------- */ + +INTERN char* cmpfilename = NULL; // name of compressed file +INTERN char* uncfilename = NULL; // name of uncompressed file +INTERN int cmpfilesize; // size of compressed file +INTERN int uncfilesize; // size of uncompressed file +INTERN int filetype; // type of current file +INTERN iostream* str_in = NULL; // input stream +INTERN iostream* str_out = NULL; // output stream + +#if !defined( BUILD_LIB ) +INTERN iostream* str_str = NULL; // storage stream + +INTERN char** filelist = NULL; // list of files to process +INTERN int file_cnt = 0; // count of files in list +INTERN int file_no = 0; // number of current file + +INTERN char** err_list = NULL; // list of error messages +INTERN int* err_tp = NULL; // list of error types +#endif + + +/* ----------------------------------------------- + global variables: messages + ----------------------------------------------- */ + +INTERN char errormessage [ 128 ]; +INTERN bool (*errorfunction)(); +INTERN int errorlevel; +// meaning of errorlevel: +// -1 -> wrong input +// 0 -> no error +// 1 -> warning +// 2 -> fatal error + + +/* ----------------------------------------------- + global variables: settings + ----------------------------------------------- */ + +#if !defined( BUILD_LIB ) +INTERN int verbosity = -1; // level of verbosity +INTERN bool overwrite = false; // overwrite files yes / no +INTERN bool wait_exit = true; // pause after finished yes / no +INTERN bool force_enc = false; // force encoding even for already compressed files yes / no +INTERN bool store_name = true; // store file name in compressed files yes / no +INTERN int verify_lv = 0; // verification level ( none (0), simple (1), detailed output (2) ) +INTERN int err_tol = 1; // error threshold ( proceed on warnings yes (2) / no (1) ) + +INTERN bool developer = false; // allow developers functions yes/no +INTERN int action = A_COMPRESS; // what to do with files + +INTERN FILE* msgout = stdout; // stream for output of messages +INTERN bool pipe_on = false; // use stdin/stdout instead of filelist +#else +INTERN bool force_enc = false; // force encoding even for already compressed files yes / no +INTERN bool store_name = false; // store file name in compressed files yes / no +INTERN int err_tol = 1; // error threshold ( proceed on warnings yes (2) / no (1) ) +INTERN int action = A_COMPRESS; // what to do with files +#endif + +/* ----------------------------------------------- + global variables: coding parameters + ----------------------------------------------- */ + +INTERN int model_order = -1; // order of statistical model (decided by analysis) +INTERN int stored_order = -1; // stored model order + + +/* ----------------------------------------------- + global variables: info about program + ----------------------------------------------- */ + +INTERN const unsigned char appversion = 6; +INTERN const char* subversion = "f"; +INTERN const char* apptitle = "packARI"; +INTERN const char* appname = "packARI"; +INTERN const char* versiondate = "01/22/2016"; +INTERN const char* author = "Matthias Stirner"; +#if !defined(BUILD_LIB) +INTERN const char* website = "http://packjpg.encode.ru/"; +INTERN const char* email = "packjpg (at) matthiasstirner.com"; +INTERN const char* copyright = "2012-2016 Matthias Stirner & HTW Aalen"; +INTERN const char* cmp_ext = "par"; +INTERN const char* unc_ext = "unc"; +#endif +INTERN const char cmp_magic[] = { 'A', 'S' }; + + +/* ----------------------------------------------- + main-function + ----------------------------------------------- */ + +#if !defined(BUILD_LIB) +int main( int argc, char** argv ) +{ + sprintf( errormessage, "no errormessage specified" ); + + clock_t begin, end; + + int error_cnt = 0; + int warn_cnt = 0; + + double acc_uncsize = 0; + double acc_cmpsize = 0; + + int kbps; + double cr; + double total; + + errorlevel = 0; + + + // read options from command line + initialize_options( argc, argv ); + + // write program info to screen + fprintf( msgout, "\n--> %s v%i.%i%s (%s) by %s <--\n", + apptitle, appversion / 10, appversion % 10, subversion, versiondate, author ); + fprintf( msgout, "Copyright %s\nAll rights reserved\n\n", copyright ); + + // check if user input is wrong, show help screen if it is + if ( ( file_cnt == 0 ) || + ( ( !developer ) && ( (action != A_COMPRESS) || (verify_lv > 1) ) ) ) { + show_help(); + return -1; + } + + // (re)set program has to be done first + reset_buffers(); + + // process file(s) - this is the main function routine + begin = clock(); + for ( file_no = 0; file_no < file_cnt; file_no++ ) { + // process current file + process_ui(); + // store error message and type if any + if ( errorlevel > 0 ) { + err_list[ file_no ] = (char*) calloc( MSG_SIZE, sizeof( char ) ); + err_tp[ file_no ] = errorlevel; + if ( err_list[ file_no ] != NULL ) + strcpy( err_list[ file_no ], errormessage ); + } + // count errors / warnings / file sizes + if ( errorlevel >= err_tol ) error_cnt++; + else { + if ( errorlevel == 1 ) warn_cnt++; + acc_uncsize += uncfilesize; + acc_cmpsize += cmpfilesize; + } + } + end = clock(); + + // errors summary: only needed for -v2 or progress bar + if ( ( verbosity == -1 ) || ( verbosity == 2 ) ) { + // print summary of errors to screen + if ( error_cnt > 0 ) { + fprintf( stderr, "\n\nfiles with errors:\n" ); + fprintf( stderr, "------------------\n" ); + for ( file_no = 0; file_no < file_cnt; file_no++ ) { + if ( err_tp[ file_no ] >= err_tol ) { + fprintf( stderr, "%s (%s)\n", filelist[ file_no ], err_list[ file_no ] ); + } + } + } + // print summary of warnings to screen + if ( warn_cnt > 0 ) { + fprintf( stderr, "\n\nfiles with warnings:\n" ); + fprintf( stderr, "------------------\n" ); + for ( file_no = 0; file_no < file_cnt; file_no++ ) { + if ( err_tp[ file_no ] == 1 ) { + fprintf( stderr, "%s (%s)\n", filelist[ file_no ], err_list[ file_no ] ); + } + } + } + } + + // show statistics + fprintf( msgout, "\n\n-> %i file(s) processed, %i error(s), %i warning(s)\n", + file_cnt, error_cnt, warn_cnt ); + if ( ( file_cnt > error_cnt ) && ( verbosity != 0 ) && + ( action == A_COMPRESS ) ) { + acc_uncsize /= 1024.0; acc_cmpsize /= 1024.0; + total = (double) ( end - begin ) / CLOCKS_PER_SEC; + kbps = ( total > 0 ) ? ( acc_uncsize / total ) : acc_uncsize; + cr = ( acc_uncsize > 0 ) ? ( 100.0 * acc_cmpsize / acc_uncsize ) : 0; + + fprintf( msgout, " -------------------------------- \n" ); + if ( total >= 0 ) { + fprintf( msgout, " total time : %8.2f sec\n", total ); + fprintf( msgout, " avrg. kbyte per s : %8i kbps\n", kbps ); + } + else { + fprintf( msgout, " total time : %8s sec\n", "N/A" ); + fprintf( msgout, " avrg. kbyte per s : %8s kbps\n", "N/A" ); + } + fprintf( msgout, " avrg. comp. ratio : %8.2f %%\n", cr ); + fprintf( msgout, " -------------------------------- \n" ); + } + + // pause before exit + if ( wait_exit && ( msgout != stderr ) ) { + fprintf( msgout, "\n\n< press ENTER >\n" ); + fgetc( stdin ); + } + + + return 0; +} +#endif + +/* ----------------------- Begin of library only functions -------------------------- */ + +/* ----------------------------------------------- + DLL export converter function + ----------------------------------------------- */ + +#if defined(BUILD_LIB) +EXPORT bool parlib_convert_stream2stream( char* msg ) +{ + // process in main function + return parlib_convert_stream2mem( NULL, NULL, msg ); +} +#endif + + +/* ----------------------------------------------- + DLL export converter function + ----------------------------------------------- */ + +#if defined(BUILD_LIB) +EXPORT bool parlib_convert_file2file( char* in, char* out, char* msg ) +{ + // init streams + parlib_init_streams( (void*) in, 0, 0, (void*) out, 0 ); + + // process in main function + return parlib_convert_stream2mem( NULL, NULL, msg ); +} +#endif + + +/* ----------------------------------------------- + DLL export converter function + ----------------------------------------------- */ + +#if defined(BUILD_LIB) +EXPORT bool parlib_convert_stream2mem( unsigned char** out_file, unsigned int* out_size, char* msg ) +{ + clock_t begin, end; + int total; + float cr; + + + // (re)set buffers + reset_buffers(); + action = A_COMPRESS; + + // main compression / decompression routines + begin = clock(); + + // process one file + process_file(); + + // fetch pointer and size of output (only for memory output) + if ( ( errorlevel < err_tol ) && ( lib_out_type == 1 ) && + ( out_file != NULL ) && ( out_size != NULL ) ) { + *out_size = str_out->getsize(); + *out_file = str_out->getptr(); + } + + // close iostreams + if ( str_in != NULL ) delete( str_in ); str_in = NULL; + if ( str_out != NULL ) delete( str_out ); str_out = NULL; + + end = clock(); + + // copy errormessage / remove files if error (and output is file) + if ( errorlevel >= err_tol ) { + if ( lib_out_type == 0 ) { + if ( filetype == F_UNK ) { + if ( file_exists( cmpfilename ) ) remove( cmpfilename ); + } else if ( filetype == F_PAR ) { + if ( file_exists( uncfilename ) ) remove( uncfilename ); + } + } + if ( msg != NULL ) strcpy( msg, errormessage ); + return false; + } + + // get compression info + total = (int) ( (double) (( end - begin ) * 1000) / CLOCKS_PER_SEC ); + cr = ( uncfilesize > 0 ) ? ( 100.0 * cmpfilesize / uncfilesize ) : 0; + + // write success message else + if ( msg != NULL ) { + switch( filetype ) + { + case F_UNK: + sprintf( msg, "Compressed to %s (%.2f%%) in %ims", + cmpfilename, cr, ( total >= 0 ) ? total : -1 ); + break; + case F_PAR: + sprintf( msg, "Decompressed to %s (%.2f%%) in %ims", + uncfilename, cr, ( total >= 0 ) ? total : -1 ); + break; + } + } + + + return true; +} +#endif + + +/* ----------------------------------------------- + DLL export init input (file/mem) + ----------------------------------------------- */ + +#if defined(BUILD_LIB) +EXPORT void parlib_init_streams( void* in_src, int in_type, int in_size, void* out_dest, int out_type ) +{ + /* a short reminder about input/output stream types: + + if input is file + ---------------- + in_scr -> name of input file + in_type -> 0 + in_size -> ignore + + if input is memory + ------------------ + in_scr -> array containg data + in_type -> 1 + in_size -> size of data array + + if input is *FILE (f.e. stdin) + ------------------------------ + in_src -> stream pointer + in_type -> 2 + in_size -> ignore + + vice versa for output streams! */ + + unsigned char buffer[ 3 ]; + + + // (re)set errorlevel + filetype = F_UNK; + errorfunction = NULL; + errorlevel = 0; + uncfilesize = 0; + cmpfilesize = 0; + + // open input stream, check for errors + str_in = new iostream( in_src, in_type, in_size, 0 ); + if ( str_in->chkerr() ) { + sprintf( errormessage, "error opening input stream" ); + errorlevel = 2; + return; + } + + // open output stream, check for errors + str_out = new iostream( out_dest, out_type, 0, 1 ); + if ( str_out->chkerr() ) { + sprintf( errormessage, "error opening output stream" ); + errorlevel = 2; + return; + } + + // free memory from filenames if needed + if ( uncfilename != NULL ) free( uncfilename ); uncfilename = NULL; + if ( cmpfilename != NULL ) free( cmpfilename ); cmpfilename = NULL; + + // check input stream + if ( !force_enc ) { + str_in->read( buffer, 1, 3 ); + if ( (buffer[0] == cmp_magic[0]) && (buffer[1] == cmp_magic[1]) ) { + // file might be PAR - check version and header + if ( check_par_header() ) { + filetype = F_PAR; + if ( buffer[2] != appversion ) { + sprintf( errormessage, "incompatible file, use %s v%i.%i", + appname, buffer[2] / 10, buffer[2] % 10 ); + errorlevel = 2; + return; + } + // copy filenames + cmpfilename = (char*) calloc( ( in_type == 0 ) ? strlen( (char*) in_src ) + 1 : 32, sizeof( char ) ); + uncfilename = (char*) calloc( ( out_type == 0 ) ? strlen( (char*) out_dest ) + 1 : 32, sizeof( char ) ); + strcpy( cmpfilename, ( in_type == 0 ) ? (char*) in_src : "PAR in memory" ); + strcpy( uncfilename, ( out_type == 0 ) ? (char*) out_dest : "ANY in memory" ); + } + } + str_in->rewind(); + } + + if ( filetype == F_UNK ) { + // file is any other file type + filetype = F_UNK; + // copy filenames + uncfilename = (char*) calloc( ( in_type == 0 ) ? strlen( (char*) in_src ) + 1 : 32, sizeof( char ) ); + cmpfilename = (char*) calloc( ( out_type == 0 ) ? strlen( (char*) out_dest ) + 1 : 32, sizeof( char ) ); + strcpy( uncfilename, ( in_type == 0 ) ? (char*) in_src : "PAR in memory" ); + strcpy( cmpfilename, ( out_type == 0 ) ? (char*) out_dest : "ANY in memory" ); + // get uncompressed file size + uncfilesize = str_in->getsize(); + str_in->rewind(); + } + + // store types of in-/output + lib_in_type = in_type; + lib_out_type = out_type; +} +#endif + + +/* ----------------------------------------------- + DLL export force encoding setting + ----------------------------------------------- */ + +#if defined(BUILD_LIB) +EXPORT void parlib_force_encoding( bool setting ) +{ + // adopt the setting + force_enc = setting; +} +#endif + + +/* ----------------------------------------------- + DLL export version information + ----------------------------------------------- */ + +#if defined(BUILD_LIB) +EXPORT const char* parlib_version_info( void ) +{ + static char v_info[ 256 ]; + + // copy version info to string + sprintf( v_info, "--> %s library v%i.%i%s (%s) by %s <--", + apptitle, appversion / 10, appversion % 10, subversion, versiondate, author ); + + return (const char*) v_info; +} +#endif + + +/* ----------------------------------------------- + DLL export version information + ----------------------------------------------- */ + +#if defined(BUILD_LIB) +EXPORT const char* parlib_short_name( void ) +{ + static char v_name[ 256 ]; + + // copy version info to string + sprintf( v_name, "%s v%i.%i%s", + apptitle, appversion / 10, appversion % 10, subversion ); + + return (const char*) v_name; +} +#endif + +/* ----------------------- End of libary only functions -------------------------- */ + +/* ----------------------- Begin of main interface functions -------------------------- */ + + +/* ----------------------------------------------- + reads in commandline arguments + ----------------------------------------------- */ + +#if !defined(BUILD_LIB) +INTERN void initialize_options( int argc, char** argv ) +{ + int tmp_val; + char** tmp_flp; + int i; + + + // get memory for filelist & preset with NULL + filelist = (char**) calloc( argc, sizeof( char* ) ); + for ( i = 0; i < argc; i++ ) + filelist[ i ] = NULL; + + // preset temporary filelist pointer + tmp_flp = filelist; + + + // read in arguments + while ( --argc > 0 ) { + argv++; + // switches begin with '-' + if ( strcmp((*argv), "-p" ) == 0 ) { + err_tol = 2; + } + else if ( strcmp((*argv), "-ver" ) == 0 ) { + verify_lv = ( verify_lv < 1 ) ? 1 : verify_lv; + } + else if ( sscanf( (*argv), "-v%i", &tmp_val ) == 1 ){ + verbosity = tmp_val; + verbosity = ( verbosity < 0 ) ? 0 : verbosity; + verbosity = ( verbosity > 2 ) ? 2 : verbosity; + } + else if ( strcmp((*argv), "-vp" ) == 0 ) { + verbosity = -1; + } + else if ( strcmp((*argv), "-np" ) == 0 ) { + wait_exit = false; + } + else if ( strcmp((*argv), "-o" ) == 0 ) { + overwrite = true; + } + else if ( strcmp((*argv), "-f" ) == 0 ) { + force_enc = true; + } + #if defined(DEV_BUILD) + else if ( strcmp((*argv), "-dev") == 0 ) { + developer = true; + } + else if ( strcmp((*argv), "-test") == 0 ) { + verify_lv = 2; + } + else if ( strcmp((*argv), "-nfn") == 0 ) { + store_name = false; + } + else if ( sscanf( (*argv), "-mo%i", &tmp_val ) == 1 ) { + tmp_val = ( tmp_val < 0 ) ? 0 : tmp_val; + tmp_val = ( tmp_val > MAX_ORDER ) ? MAX_ORDER : tmp_val; + stored_order = tmp_val; + } + #endif + else if ( strcmp((*argv), "-") == 0 ) { + // switch standard message out stream + msgout = stderr; + // use "-" as placeholder for stdin + *(tmp_flp++) = (char*) "-"; + } + else { + // if argument is not switch, it's a filename + *(tmp_flp++) = *argv; + } + } + + // count number of files (or filenames) in filelist + for ( file_cnt = 0; filelist[ file_cnt ] != NULL; file_cnt++ ); + + // alloc arrays for error messages and types storage + err_list = (char**) calloc( file_cnt, sizeof( char* ) ); + err_tp = (int*) calloc( file_cnt, sizeof( int ) ); +} +#endif + + +/* ----------------------------------------------- + UI for processing one file + ----------------------------------------------- */ + +#if !defined(BUILD_LIB) +INTERN void process_ui( void ) +{ + clock_t begin, end; + const char* actionmsg = NULL; + const char* errtypemsg = NULL; + int total, bpms; + float cr; + + + errorfunction = NULL; + errorlevel = 0; + uncfilesize = 0; + cmpfilesize = 0; + #if !defined(DEV_BUILD) + action = A_COMPRESS; + #endif + + // compare file name, set pipe if needed + if ( ( strcmp( filelist[ file_no ], "-" ) == 0 ) && ( action == A_COMPRESS ) ) { + pipe_on = true; + filelist[ file_no ] = (char*) "STDIN"; + } + else { + pipe_on = false; + } + + if ( verbosity >= 0 ) { // standard UI + fprintf( msgout, "\nProcessing file %i of %i \"%s\" -> ", + file_no + 1, file_cnt, filelist[ file_no ] ); + + if ( verbosity > 1 ) + fprintf( msgout, "\n----------------------------------------" ); + + // check input file and determine filetype + execute( check_file ); + + // get specific action message + switch ( action ) { + case A_COMPRESS: ( filetype == F_UNK ) ? actionmsg = "Compressing" : actionmsg = "Decompressing"; + break; + } + + if ( verbosity < 2 ) fprintf( msgout, "%s -> ", actionmsg ); + } + else { // progress bar UI + // update progress message + fprintf( msgout, "Processing file %2i of %2i ", file_no + 1, file_cnt ); + progress_bar( file_no, file_cnt ); + fprintf( msgout, "\r" ); + execute( check_file ); + } + fflush( msgout ); + + + // main function routine + begin = clock(); + + // streams are initiated, start processing file + process_file(); + + // close iostreams + if ( str_in != NULL ) delete( str_in ); str_in = NULL; + if ( str_out != NULL ) delete( str_out ); str_out = NULL; + if ( str_str != NULL ) delete( str_str ); str_str = NULL; + // delete if broken or if output not needed + if ( ( !pipe_on ) && ( ( errorlevel >= err_tol ) || ( action != A_COMPRESS ) ) ) { + if ( filetype == F_UNK ) { + if ( file_exists( cmpfilename ) ) remove( cmpfilename ); + } else if ( filetype == F_PAR ) { + if ( file_exists( uncfilename ) ) remove( uncfilename ); + } + } + + end = clock(); + + // speed and compression ratio calculation + total = (int) ( (double) (( end - begin ) * 1000) / CLOCKS_PER_SEC ); + bpms = ( total > 0 ) ? ( uncfilesize / total ) : uncfilesize; + cr = ( uncfilesize > 0 ) ? ( 100.0 * cmpfilesize / uncfilesize ) : 0; + + + if ( verbosity >= 0 ) { // standard UI + if ( verbosity > 1 ) + fprintf( msgout, "\n----------------------------------------" ); + + // display success/failure message + switch ( verbosity ) { + case 0: + if ( errorlevel < err_tol ) { + if ( action == A_COMPRESS ) fprintf( msgout, "%.2f%%", cr ); + else fprintf( msgout, "DONE" ); + } + else fprintf( msgout, "ERROR" ); + if ( errorlevel > 0 ) fprintf( msgout, "\n" ); + break; + + case 1: + fprintf( msgout, "%s\n", ( errorlevel < err_tol ) ? "DONE" : "ERROR" ); + break; + + case 2: + if ( errorlevel < err_tol ) fprintf( msgout, "\n-> %s OK\n", actionmsg ); + else fprintf( msgout, "\n-> %s ERROR\n", actionmsg ); + break; + } + + // set type of error message + switch ( errorlevel ) { + case 0: errtypemsg = "none"; break; + case 1: ( err_tol > 1 ) ? errtypemsg = "warning (ignored)" : errtypemsg = "warning (skipped file)"; break; + case 2: errtypemsg = "fatal error"; break; + } + + // error/ warning message + if ( errorlevel > 0 ) { + fprintf( msgout, " %s -> %s:\n", get_status( errorfunction ), errtypemsg ); + fprintf( msgout, " %s\n", errormessage ); + } + if ( (verbosity > 0) && (errorlevel < err_tol) && (action == A_COMPRESS) ) { + if ( total >= 0 ) { + fprintf( msgout, " time taken : %7i msec\n", total ); + fprintf( msgout, " byte per ms : %7i byte\n", bpms ); + } + else { + fprintf( msgout, " time taken : %7s msec\n", "N/A" ); + fprintf( msgout, " byte per ms : %7s byte\n", "N/A" ); + } + fprintf( msgout, " comp. ratio : %7.2f %%\n", cr ); + } + if ( ( verbosity > 1 ) && ( action == A_COMPRESS ) ) + fprintf( msgout, "\n" ); + } + else { // progress bar UI + // if this is the last file, update progress bar one last time + if ( file_no + 1 == file_cnt ) { + // update progress message + fprintf( msgout, "Processed %2i of %2i files ", file_no + 1, file_cnt ); + progress_bar( 1, 1 ); + fprintf( msgout, "\r" ); + } + } +} +#endif + + +/* ----------------------------------------------- + gets statusmessage for function + ----------------------------------------------- */ + +#if !defined(BUILD_LIB) +INTERN inline const char* get_status( bool (*function)() ) +{ + if ( function == NULL ) { + return "unknown action"; + } else if ( function == *check_file ) { + return "Determining filetype"; + } else if ( function == *decide_model_order ) { + return "Deciding model order"; + } else if ( function == *encode_file ) { + return "Compressing file"; + } else if ( function == *decode_file ) { + return "Decompressing file"; + } else if ( function == *swap_streams ) { + return "Swapping input/output streams"; + } else if ( function == *compare_output ) { + return "Verifying output stream"; + } else if ( function == *reset_buffers ) { + return "Resetting program"; + } + #if defined(DEV_BUILD) + // no specific developer functions! + #endif + else { + return "Function description missing!"; + } +} +#endif + + +/* ----------------------------------------------- + shows help in case of wrong input + ----------------------------------------------- */ + +#if !defined(BUILD_LIB) +INTERN void show_help( void ) +{ + fprintf( msgout, "\n" ); + fprintf( msgout, "Website: %s\n", website ); + fprintf( msgout, "Email : %s\n", email ); + fprintf( msgout, "\n" ); + fprintf( msgout, "Usage: %s [switches] [filename(s)]", appname ); + fprintf( msgout, "\n" ); + fprintf( msgout, "\n" ); + fprintf( msgout, " [-f] force encoding for compressed files\n" ); + fprintf( msgout, " [-ver] verify files after processing\n" ); + fprintf( msgout, " [-v?] set level of verbosity (max: 2) (def: 0)\n" ); + fprintf( msgout, " [-np] no pause after processing files\n" ); + fprintf( msgout, " [-o] overwrite existing files\n" ); + fprintf( msgout, " [-p] proceed on warnings\n" ); + #if defined(DEV_BUILD) + if ( developer ) { + fprintf( msgout, "\n" ); + fprintf( msgout, " [-mo?] force model order for coding (0...%i)\n", MAX_ORDER ); + fprintf( msgout, " [-nfn] don't store filenames\n" ); + } + #endif + fprintf( msgout, "\n" ); + fprintf( msgout, "Examples: \"%s -v1 -o image??.%s\"\n", appname, cmp_ext ); + fprintf( msgout, " \"%s -p *.txt\"\n", appname ); +} +#endif + + +/* ----------------------------------------------- + processes one file + ----------------------------------------------- */ + +INTERN void process_file( void ) +{ + if ( filetype == F_UNK ) { + switch ( action ) { + case A_COMPRESS: + execute( decide_model_order ); + execute( encode_file ); + #if !defined(BUILD_LIB) + if ( verify_lv > 0 ) { // verifcation + execute( reset_buffers ); + execute( swap_streams ); + execute( decode_file ); + execute( compare_output ); + } + #endif + break; + + #if !defined(BUILD_LIB) && defined(DEV_BUILD) + // no developers functionality! + #else + default: + break; + #endif + } + } + else if ( filetype == F_PAR ) { + switch ( action ) + { + case A_COMPRESS: + execute( decode_file ); + #if !defined(BUILD_LIB) + // this does not work yet! + // and it's not even needed + /*if ( verify_lv > 0 ) { // verifcation + execute( reset_buffers ); + execute( swap_streams ); + execute( decide_model_order ); + execute( encode_file ); + execute( compare_output ); + }*/ + #endif + break; + + #if !defined(BUILD_LIB) && defined(DEV_BUILD) + // no developers functionality! + #else + default: + break; + #endif + } + } + #if !defined(BUILD_LIB) && defined(DEV_BUILD) + // write error file if verify lv > 1 + if ( ( verify_lv > 1 ) && ( errorlevel >= err_tol ) ) + write_errfile(); + #endif + // reset buffers + reset_buffers(); +} + + +/* ----------------------------------------------- + main-function execution routine + ----------------------------------------------- */ + +INTERN void execute( bool (*function)() ) +{ + if ( errorlevel < err_tol ) { + #if !defined BUILD_LIB + clock_t begin, end; + bool success; + int total; + + // write statusmessage + if ( verbosity == 2 ) { + fprintf( msgout, "\n%s ", get_status( function ) ); + for ( int i = strlen( get_status( function ) ); i <= 30; i++ ) + fprintf( msgout, " " ); + } + + // set starttime + begin = clock(); + // call function + success = ( *function )(); + // set endtime + end = clock(); + + if ( ( errorlevel > 0 ) && ( errorfunction == NULL ) ) + errorfunction = function; + + // write time or failure notice + if ( success ) { + total = (int) ( (double) (( end - begin ) * 1000) / CLOCKS_PER_SEC ); + if ( verbosity == 2 ) fprintf( msgout, "%6ims", ( total >= 0 ) ? total : -1 ); + } + else { + errorfunction = function; + if ( verbosity == 2 ) fprintf( msgout, "%8s", "ERROR" ); + } + #else + // call function + ( *function )(); + + // store errorfunction if needed + if ( ( errorlevel > 0 ) && ( errorfunction == NULL ) ) + errorfunction = function; + #endif + } +} + +/* ----------------------- End of main interface functions -------------------------- */ + +/* ----------------------- Begin of main functions -------------------------- */ + + +/* ----------------------------------------------- + check file and determine filetype + ----------------------------------------------- */ + +#if !defined(BUILD_LIB) +INTERN bool check_file( void ) +{ + unsigned char fileid[ 3 ] = { 0, 0, 0 }; + const char* filename = filelist[ file_no ]; + + // preset file type + filetype = F_UNK; + + // open input stream, check for errors + str_in = new iostream( (void*) filename, ( !pipe_on ) ? 0 : 2, 0, 0 ); + if ( str_in->chkerr() ) { + sprintf( errormessage, FRD_ERRMSG ); + errorlevel = 2; + return false; + } + + // free memory from filenames if needed + if ( uncfilename != NULL ) free( uncfilename ); uncfilename = NULL; + if ( cmpfilename != NULL ) free( cmpfilename ); cmpfilename = NULL; + + // check if this is a compressed par file + while ( !force_enc ) { + // read file id and version number + if ( str_in->read( fileid, 1, 3 ) != 3 ) break; + // check magic number + if ( ( fileid[0] != cmp_magic[0] ) || ( fileid[1] != cmp_magic[1] ) ) break; + // check par header, extract file name + if ( !check_par_header() ) break; + // alright: this is a .par file! + // compare version number + if ( fileid[2] != appversion ) { + sprintf( errormessage, "incompatible file, use %s v%i.%i", + appname, fileid[2] / 10, fileid[2] % 10 ); + errorlevel = 2; + return false; + } + // file is .par and compatible + filetype = F_PAR; + // create filenames + if ( !pipe_on ) { + cmpfilename = (char*) calloc( strlen( filename ) + 1, sizeof( char ) ); + strcpy( cmpfilename, filename ); + if ( uncfilename != NULL ) { + // ugly fix for path names in drag and drop + if ( strrchr( uncfilename, '.' ) != NULL ) { + uncfilename = ( overwrite ) ? + create_filename( filename, strrchr( uncfilename, '.' ) + 1 ) : + unique_filename( filename, strrchr( uncfilename, '.' ) + 1 ); + } else { + uncfilename = ( overwrite ) ? + create_filename( filename, NULL ) : + unique_filename( filename, NULL ); + } + // also, making full file name redundant + /*if ( !overwrite ) while ( file_exists( uncfilename ) ) { + uncfilename = (char*) realloc( uncfilename, strlen( uncfilename ) + 2 ); + add_underscore( uncfilename ); + }*/ + } else { + uncfilename = ( overwrite ) ? + create_filename( filename, (char*) unc_ext ) : + unique_filename( filename, (char*) unc_ext ); + } + } + else { + free( uncfilename ); + uncfilename = create_filename( "STDOUT", NULL ); + cmpfilename = create_filename( "STDIN", NULL ); + } + // open output stream, check for errors + str_out = new iostream( (void*) uncfilename, ( !pipe_on ) ? 0 : 2, 0, 1 ); + if ( str_out->chkerr() ) { + sprintf( errormessage, FWR_ERRMSG ); + errorlevel = 2; + return false; + } + break; + } + + // filetype is unknown, or compression is forced + if ( filetype == F_UNK ) { + // free uncompressed file name again + if ( uncfilename != NULL ) free( uncfilename ); uncfilename = NULL; + // create filenames + if ( !pipe_on ) { + uncfilename = (char*) calloc( strlen( filename ) + 1, sizeof( char ) ); + strcpy( uncfilename, filename ); + cmpfilename = ( overwrite ) ? + create_filename( filename, (char*) cmp_ext ) : + unique_filename( filename, (char*) cmp_ext ); + } + else { + uncfilename = create_filename( "STDIN", NULL ); + cmpfilename = create_filename( "STDOUT", NULL ); + } + // open output stream, check for errors + str_out = new iostream( (void*) cmpfilename, ( !pipe_on ) ? 0 : 2, 0, 1 ); + if ( str_out->chkerr() ) { + sprintf( errormessage, FWR_ERRMSG ); + errorlevel = 2; + return false; + } + // get uncompressed file size + uncfilesize = str_in->getsize(); + } + + // rewind (need to start from the beginning) + if ( str_in->rewind() != 0 ) { + sprintf( errormessage, FRD_ERRMSG ); + errorlevel = 2; + return false; + } + + + return true; +} +#endif + + +/* ----------------------------------------------- + swap streams / init verification + ----------------------------------------------- */ +#if !defined(BUILD_LIB) +INTERN bool swap_streams( void ) +{ + // store input stream + str_str = str_in; + str_str->rewind(); + + // replace input stream by output stream / switch mode for reading + str_in = str_out; + str_in->switch_mode(); + + // open new stream for output / check for errors + str_out = new iostream( NULL, 1, 0, 1 ); + if ( str_out->chkerr() ) { + sprintf( errormessage, "error opening comparison stream" ); + errorlevel = 2; + return false; + } + + + return true; +} +#endif + + +/* ----------------------------------------------- + comparison between input & output + ----------------------------------------------- */ +#if !defined( BUILD_LIB ) +INTERN bool compare_output( void ) +{ + unsigned char* buff_ori; + unsigned char* buff_cmp; + int bsize = 1024; + int dsize; + int i, b; + + + // init buffer arrays + buff_ori = ( unsigned char* ) calloc( bsize, sizeof( char ) ); + buff_cmp = ( unsigned char* ) calloc( bsize, sizeof( char ) ); + if ( ( buff_ori == NULL ) || ( buff_cmp == NULL ) ) { + if ( buff_ori != NULL ) free( buff_ori ); + if ( buff_cmp != NULL ) free( buff_cmp ); + sprintf( errormessage, MEM_ERRMSG ); + errorlevel = 2; + return false; + } + + // switch output stream mode / check for stream errors + str_out->switch_mode(); + while ( true ) { + if ( str_out->chkerr() ) + sprintf( errormessage, "error in comparison stream" ); + else if ( str_in->chkerr() ) + sprintf( errormessage, "error in output stream" ); + else if ( str_str->chkerr() ) + sprintf( errormessage, "error in input stream" ); + else break; + errorlevel = 2; + return false; + } + + // compare sizes + dsize = str_str->getsize(); + /*if ( str_out->getsize() != dsize ) { + sprintf( errormessage, "file sizes do not match" ); + errorlevel = 2; + return false; + }*/ + + // compare files byte by byte + for ( i = 0; i < dsize; i++ ) { + b = i % bsize; + if ( b == 0 ) { + str_str->read( buff_ori, sizeof( char ), bsize ); + str_out->read( buff_cmp, sizeof( char ), bsize ); + } + if ( buff_ori[ b ] != buff_cmp[ b ] ) { + sprintf( errormessage, "difference found at 0x%X", i ); + errorlevel = 2; + return false; + } + } + + + return true; +} +#endif + + +/* ----------------------------------------------- + set each variable to its initial value + ----------------------------------------------- */ + +INTERN bool reset_buffers( void ) +{ + // --- restore original settings --- + model_order = stored_order; + + + return true; +} + + +/* ----------------------------------------------- + decide order for statistical model + ----------------------------------------------- */ + +INTERN bool decide_model_order( void ) +{ + double cost[ MAX_ORDER + 1 ]; + unsigned char lbyte[ MAX_ORDER + 1 ]; + unsigned char byte; + model_s* model; + int o, p; + int i; + + + // calculate cost for CHECK_BYTES and each order + for ( o = 0; o <= MAX_ORDER; o++ ) { + // preparations + model = INIT_MODEL_S( 256, 256, o ); + memset( lbyte, 0, MAX_ORDER + 1 ); + str_in->rewind(); + cost[ o ] = 0; + // main testing routine + for ( p = 0; p < CHECK_BYTES; p++ ) { + if ( str_in->read( &byte, 1, 1 ) != 1 ) break; + // context shifting + for ( i = o-1; i >= 0; i-- ) + model->shift_context( lbyte[i] ); + // cost count + cost[ o ] += cost_count_ari( model, byte ); + // context update + for ( i = 1; i < o; i++ ) + lbyte[i] = lbyte[i-1]; + lbyte[0] = byte; + } + // clean up + delete( model ); + } + + // rewind (need to start from the beginning) + if ( str_in->rewind() != 0 ) { + sprintf( errormessage, FRD_ERRMSG ); + errorlevel = 2; + return false; + } + + // find the best one + if ( stored_order == -1 ) for ( model_order = 0, o = 1; o <= MAX_ORDER; o++ ) + if ( cost[ model_order ] > cost[ o ] ) model_order = o; + + // check for incompressible files + if ( (cost[ model_order ]/8) >= p ) { + sprintf( errormessage, "file seems to be incompressible" ); + errorlevel = 1; + } + + + return true; +} + + +/* ----------------------------------------------- + encode one file + ----------------------------------------------- */ + +INTERN bool encode_file( void ) +{ + // --- write PAR header --- + str_out->write( (void*) cmp_magic, 1, 2 ); + str_out->write( (void*) &appversion, 1, 1 ); + if ( !write_par_header() ) return false; + + // --- encode file --- + if ( !endecode_file( true ) ) return false; + + // --- get compressed file size --- + cmpfilesize = str_out->getsize(); + + + return true; +} + + +/* ----------------------------------------------- + decode one file + ----------------------------------------------- */ + +INTERN bool decode_file( void ) +{ + unsigned char byte; + + // --- get compressed file size --- + cmpfilesize = str_in->getsize(); + + // --- read PAR header again --- + str_in->read( (void*) &byte, 1, 1 ); // skip byte 0 + str_in->read( (void*) &byte, 1, 1 ); // skip byte 1 + str_in->read( (void*) &byte, 1, 1 ); // skip byte 2 + if ( !read_par_header() ) { + sprintf( errormessage, "this should not have happened :-(" ); + errorlevel = 2; + return false; + } + + // --- decode file --- + if ( !endecode_file( false ) ) return false; + + + return true; +} + +/* ----------------------- End of main functions -------------------------- */ + +/* ----------------------- Begin of other coding functions -------------------------- */ + + +/* ----------------------------------------------- + check par header, extract uncompressed name + ----------------------------------------------- */ +INTERN inline bool check_par_header( void ) +{ + unsigned char hash = 0x00; + unsigned char byte; + char* fn; + int i; + + + for ( i = 0; i < 5; i++ ) { + if ( str_in->read( (void*) &byte, 1, 1 ) != 1 ) return false; + hash = pearson_hash[ byte ^ hash ]; + } + + // fetch file name, followed by 0x00, do hashing + if ( str_in->read( (void*) &byte, 1, 1 ) != 1 ) return false; + if ( byte != 0x00 ) { + uncfilename = ( char* ) calloc( 256 + 1, sizeof( char ) ); // !!! + fn = uncfilename; hash = pearson_hash[ byte ^ hash ]; + for ( (*fn++) = byte, i = 1; i < 256; i++, fn++ ) { + if ( str_in->read( (void*) fn, 1, 1 ) != 1 ) return false; + hash = pearson_hash[ (*fn) ^ hash ]; + if ( (*fn) == 0x00 ) break; + } + if ( i == 256 ) return false; + } else hash = pearson_hash[ hash ]; + + // check hash + if ( str_in->read( (void*) &byte, 1, 1 ) != 1 ) return false; + if ( hash != byte ) return false; + + + // success! + return true; +} + + +/* ----------------------------------------------- + read/check header of compressed file + ----------------------------------------------- */ +INTERN inline bool read_par_header( void ) +{ + unsigned char nbyte[4] = { 0 }; + unsigned char hash = 0x00; + unsigned char byte; + int i; + + + // read model_order, do hashing + model_order = 0; + if ( str_in->read( &model_order, 1, 1 ) != 1 ) return false; + hash = pearson_hash[ model_order ^ hash ]; + + // read number of bytes, do hashing + if ( str_in->read( (void*) nbyte, 1, 4 ) != 4 ) return false; + for ( i = 0; i < 4; i++ ) hash = pearson_hash[ nbyte[i] ^ hash ]; + + // only check file name followed by 0x00, do hashing + for ( i = 0; i < 256; i++ ) { + if ( str_in->read( (void*) &byte, 1, 1 ) != 1 ) return false; + hash = pearson_hash[ byte ^ hash ]; + if ( byte == 0x00 ) break; + } + if ( i == 256 ) return false; + + // check hash + if ( str_in->read( (void*) &byte, 1, 1 ) != 1 ) return false; + if ( hash != byte ) return false; + + // convert number of bytes from little endian + uncfilesize = 0; + uncfilesize |= nbyte[0] << 24; + uncfilesize |= nbyte[1] << 16; + uncfilesize |= nbyte[2] << 8; + uncfilesize |= nbyte[3] << 0; + + + // success! + return true; +} + + +/* ----------------------------------------------- + write header for compressed file + ----------------------------------------------- */ +INTERN inline bool write_par_header( void ) +{ + unsigned char nbyte[4] = { 0 }; + unsigned char hash = 0x00; + unsigned char zero = 0x00; + char* fn; + int i; + + + // convert number of bytes to little endian + nbyte[0] = ( uncfilesize >> 24 ) & 0xFF; + nbyte[1] = ( uncfilesize >> 16 ) & 0xFF; + nbyte[2] = ( uncfilesize >> 8 ) & 0xFF; + nbyte[3] = ( uncfilesize >> 0 ) & 0xFF; + + // write coding parameters, do hashing + str_out->write( &model_order, 1, 1 ); + hash = pearson_hash[ model_order ^ hash ]; + + // write number of bytes, do hashing + str_out->write( (void*) nbyte, 1, 4 ); + for ( i = 0; i < 4; i++ ) hash = pearson_hash[ nbyte[i] ^ hash ]; + + // write uncompressed file name, do hashing + if ( store_name ) { + // get filename without path + if ( strrchr( uncfilename, '/' ) != NULL ) { + fn = strrchr( uncfilename, '/' ) + 1; + if ( strrchr( fn, '\\' ) != NULL ) + fn = strrchr( fn, '\\' ) + 1; + } else if ( strrchr( uncfilename, '\\' ) != NULL ) + fn = strrchr( uncfilename, '\\' ) + 1; + else fn = uncfilename; + // store filename, do hashing + str_out->write( (void*) fn, 1, strlen( fn ) + 1 ); + for ( ; (*fn) != 0x00; fn++ ) + hash = pearson_hash[ (*fn) ^ hash ]; + } else str_out->write( (void*) &zero, 1, 1 ); + hash = pearson_hash[ hash ]; + + // write hash + str_out->write( (void*) &hash, 1, 1 ); + + + // success! + return true; +} + + +/* ----------------------------------------------- + encode or decode file + ----------------------------------------------- */ +INTERN bool endecode_file( bool encoding ) { + // this will either encode or decode one file + // decision is based on the file type + aricoder* coder; + model_s* model; + unsigned char lbyte[ MAX_ORDER + 1 ]; + unsigned char byte; + int p, i; + + + // --- preparations --- + + // init aricoder for encoding/decoding + coder = ( encoding ) ? + new aricoder( str_out, 1 ) : new aricoder( str_in, 0 ); + + // init statistical model + model = INIT_MODEL_S( 256, 256, model_order ); + + // memset last byte + memset( lbyte, 0, MAX_ORDER + 1 ); + + + // --- main encoding/decoding routine --- + + // main routine + for ( p = 0; p < uncfilesize; p++ ) { + // context shifting + for ( i = model_order-1; i >= 0; i-- ) + model->shift_context( lbyte[i] ); + + // actual encoding / decoding + if ( encoding ) { // encoding / loop independent condition + str_in->read( &byte, 1, 1 ); + encode_ari( coder, model, byte ); + } else { // decoding + byte = decode_ari( coder, model ); + str_out->write( &byte, 1, 1 ); + } + + // context update + for ( i = 1; i < model_order; i++ ) + lbyte[i] = lbyte[i-1]; + lbyte[0] = byte; + } + + + // --- cleanup --- + + // free coder and models + delete( coder ); + delete( model ); + + + return true; +} + +/* ----------------------------------------------- + model_s cost count + ----------------------------------------------- */ +INTERN inline double cost_count_ari( model_s* model, int c ) +{ + static symbol s; + static int esc; + double cost = 0; + + do { + esc = model->convert_int_to_symbol( c, &s ); + cost -= log2( (double) (s.high_count - s.low_count) / (double) s.scale ); + } while ( esc ); + model->update_model( c ); + + return cost; +} + +/* ----------------------- End of other coding functions -------------------------- */ + +/* ----------------------- Begin of miscellaneous helper functions -------------------------- */ + + +/* ----------------------------------------------- + displays progress bar on screen + ----------------------------------------------- */ +#if !defined(BUILD_LIB) +INTERN inline void progress_bar( int current, int last ) +{ + int barpos = ( ( current * BARLEN ) + ( last / 2 ) ) / last; + int i; + + + // generate progress bar + fprintf( msgout, "[" ); + #if defined(_WIN32) + for ( i = 0; i < barpos; i++ ) + fprintf( msgout, "\xFE" ); + #else + for ( i = 0; i < barpos; i++ ) + fprintf( msgout, "X" ); + #endif + for ( ; i < BARLEN; i++ ) + fprintf( msgout, " " ); + fprintf( msgout, "]" ); +} +#endif + +/* ----------------------------------------------- + creates filename, callocs memory for it + ----------------------------------------------- */ +#if !defined(BUILD_LIB) +INTERN inline char* create_filename( const char* base, const char* extension ) +{ + int len = strlen( base ) + ( ( extension == NULL ) ? 0 : strlen( extension ) + 1 ) + 1; + char* filename = (char*) calloc( len, sizeof( char ) ); + + // create a filename from base & extension + strcpy( filename, base ); + set_extension( filename, extension ); + + return filename; +} +#endif + +/* ----------------------------------------------- + creates filename, callocs memory for it + ----------------------------------------------- */ +#if !defined(BUILD_LIB) +INTERN inline char* unique_filename( const char* base, const char* extension ) +{ + int len = strlen( base ) + ( ( extension == NULL ) ? 0 : strlen( extension ) + 1 ) + 1; + char* filename = (char*) calloc( len, sizeof( char ) ); + + // create a unique filename using underscores + strcpy( filename, base ); + set_extension( filename, extension ); + while ( file_exists( filename ) ) { + len += sizeof( char ); + filename = (char*) realloc( filename, len ); + add_underscore( filename ); + } + + return filename; +} +#endif + +/* ----------------------------------------------- + changes extension of filename + ----------------------------------------------- */ +#if !defined(BUILD_LIB) +INTERN inline void set_extension( const char* filename, const char* extension ) +{ + char* extstr; + + // find position of extension in filename + extstr = ( strrchr( filename, '.' ) == NULL ) ? + strrchr( filename, '\0' ) : strrchr( filename, '.' ); + + // set new extension + if ( extension != NULL ) { + (*extstr++) = '.'; + strcpy( extstr, extension ); + } + else + (*extstr) = '\0'; +} +#endif + +/* ----------------------------------------------- + adds underscore after filename + ----------------------------------------------- */ +#if !defined(BUILD_LIB) +INTERN inline void add_underscore( char* filename ) +{ + char* tmpname = (char*) calloc( strlen( filename ) + 1, sizeof( char ) ); + char* extstr; + + // copy filename to tmpname + strcpy( tmpname, filename ); + // search extension in filename + extstr = strrchr( filename, '.' ); + + // add underscore before extension + if ( extstr != NULL ) { + (*extstr++) = '_'; + strcpy( extstr, strrchr( tmpname, '.' ) ); + } + else + sprintf( filename, "%s_", tmpname ); + + // free memory + free( tmpname ); +} +#endif + +/* ----------------------------------------------- + checks if a file exists + ----------------------------------------------- */ +INTERN inline bool file_exists( const char* filename ) +{ + // needed for both, executable and library + FILE* fp = fopen( filename, "rb" ); + + if ( fp == NULL ) return false; + else { + fclose( fp ); + return true; + } +} + +/* ----------------------- End of miscellaneous helper functions -------------------------- */ + +/* ----------------------- Begin of developers functions -------------------------- */ + + +#if !defined(BUILD_LIB) && defined(DEV_BUILD) +/* ----------------------------------------------- + Writes error info file + ----------------------------------------------- */ +INTERN bool write_errfile( void ) +{ + FILE* fp; + char* fn; + + + // return immediately if theres no error + if ( errorlevel == 0 ) return true; + + // create filename based on errorlevel + if ( errorlevel == 1 ) { + fn = create_filename( filelist[ file_no ], "wrn.nfo" ); + } + else { + fn = create_filename( filelist[ file_no ], "err.nfo" ); + } + + // open file for output + fp = fopen( fn, "w" ); + if ( fp == NULL ){ + sprintf( errormessage, FWR_ERRMSG ); + errorlevel = 2; + return false; + } + free( fn ); + + // write status and errormessage to file + fprintf( fp, "--> error (level %i) in file \"%s\" <--\n", errorlevel, filelist[ file_no ] ); + fprintf( fp, "\n" ); + // write error specification to file + fprintf( fp, " %s -> %s:\n", get_status( errorfunction ), + ( errorlevel == 1 ) ? "warning" : "error" ); + fprintf( fp, " %s\n", errormessage ); + + // done, close file + fclose( fp ); + + + return true; +} +#endif + +/* ----------------------- End of developers functions -------------------------- */ + +/* ----------------------- End of file -------------------------- */ diff --git a/contrib/preflate/packARI/source/packaridll.h b/contrib/preflate/packARI/source/packaridll.h new file mode 100644 index 0000000..26f654a --- /dev/null +++ b/contrib/preflate/packARI/source/packaridll.h @@ -0,0 +1,37 @@ +// packARIdll.h - function import declarations for the packARI DLL +#define IMPORT __declspec( dllimport ) + +/* ----------------------------------------------- + function declarations: library only functions + ----------------------------------------------- */ + +IMPORT bool parlib_convert_stream2stream( char* msg ); +IMPORT bool parlib_convert_file2file( char* in, char* out, char* msg ); +IMPORT bool parlib_convert_stream2mem( unsigned char** out_file, unsigned int* out_size, char* msg ); +IMPORT void parlib_init_streams( void* in_src, int in_type, int in_size, void* out_dest, int out_type ); +IMPORT void parlib_force_encoding( bool setting ); +IMPORT const char* parlib_version_info( void ); +IMPORT const char* parlib_short_name( void ); + +/* a short reminder about input/output stream types + for the parlib_init_streams() function + + if input is file + ---------------- + in_scr -> name of input file + in_type -> 0 + in_size -> ignore + + if input is memory + ------------------ + in_scr -> array containg data + in_type -> 1 + in_size -> size of data array + + if input is *FILE (f.e. stdin) + ------------------------------ + in_src -> stream pointer + in_type -> 2 + in_size -> ignore + + vice versa for output streams! */ diff --git a/contrib/preflate/packARI/source/packarilib.h b/contrib/preflate/packARI/source/packarilib.h new file mode 100644 index 0000000..caca6ad --- /dev/null +++ b/contrib/preflate/packARI/source/packarilib.h @@ -0,0 +1,41 @@ +// packARIlib.h - function declarations for the packARI library +#if defined BUILD_DLL + #define EXPORT __declspec( dllexport ) +#else + #define EXPORT extern +#endif + +/* ----------------------------------------------- + function declarations: library only functions + ----------------------------------------------- */ + +EXPORT bool parlib_convert_stream2stream( char* msg ); +EXPORT bool parlib_convert_file2file( char* in, char* out, char* msg ); +EXPORT bool parlib_convert_stream2mem( unsigned char** out_file, unsigned int* out_size, char* msg ); +EXPORT void parlib_init_streams( void* in_src, int in_type, int in_size, void* out_dest, int out_type ); +EXPORT void parlib_force_encoding( bool setting ); +EXPORT const char* parlib_version_info( void ); +EXPORT const char* parlib_short_name( void ); + +/* a short reminder about input/output stream types + for the parlib_init_streams() function + + if input is file + ---------------- + in_scr -> name of input file + in_type -> 0 + in_size -> ignore + + if input is memory + ------------------ + in_scr -> array containg data + in_type -> 1 + in_size -> size of data array + + if input is *FILE (f.e. stdin) + ------------------------------ + in_src -> stream pointer + in_type -> 2 + in_size -> ignore + + vice versa for output streams! */ diff --git a/contrib/preflate/packARI/source/paritbl.h b/contrib/preflate/packARI/source/paritbl.h new file mode 100644 index 0000000..da8cff9 --- /dev/null +++ b/contrib/preflate/packARI/source/paritbl.h @@ -0,0 +1,48 @@ +/* ----------------------------------------------- + setting for packARI + ----------------------------------------------- */ + +// !EDIT WITH CAUTION! +// each change may lead to incompatibilities with previous version + +// action defines +#define A_COMPRESS 1 + +// file type defines +#define F_UNK 1 +#define F_PAR 2 + +// compression parameters (tweaking possible) +#define MAX_ORDER 3 +#define CHECK_BYTES ( 1 << 14 ) + +// other setting +#define ANALYSIS_CSV "par_analysis.csv" + + +/* ----------------------------------------------- + pearson hashing table + ----------------------------------------------- */ + +// copied from Table 1 in: +// Pearson, Peter K. (June 1990), "Fast Hashing of Variable-Length Text Strings", +// Communications of the ACM 33 (6): 677, doi:10.1145/78973.78978 +static int pearson_hash[ 256 ] = +{ + 1, 87, 49, 12, 176, 178, 102, 166, 121, 193, 6, 84, 249, 230, 44, 163, + 14, 197, 213, 181, 161, 85, 218, 80, 64, 239, 24, 226, 236, 142, 38, 200, + 110, 177, 104, 103, 141, 253, 255, 50, 77, 101, 81, 18, 45, 96, 31, 222, + 25, 107, 190, 70, 86, 237, 240, 34, 72, 242, 20, 214, 244, 227, 149, 235, + 97, 234, 57, 22, 60, 250, 82, 175, 208, 5, 127, 199, 111, 62, 135, 248, + 174, 169, 211, 58, 66, 154, 106, 195, 245, 171, 17, 187, 182, 179, 0, 243, + 132, 56, 148, 75, 128, 133, 158, 100, 130, 126, 91, 13, 153, 246, 216, 219, + 119, 68, 223, 78, 83, 88, 201, 99, 122, 11, 92, 32, 136, 114, 52, 10, + 138, 30, 48, 183, 156, 35, 61, 26, 143, 74, 251, 94, 129, 162, 63, 152, + 170, 7, 115, 167, 241, 206, 3, 150, 55, 59, 151, 220, 90, 53, 23, 131, + 125, 173, 15, 238, 79, 95, 89, 16, 105, 137, 225, 224, 217, 160, 37, 123, + 118, 73, 2, 157, 46, 116, 9, 145, 134, 228, 207, 212, 202, 215, 69, 229, + 27, 188, 67, 124, 168, 252, 42, 4, 29, 108, 21, 247, 19, 205, 39, 203, + 233, 40, 186, 147, 198, 192, 155, 33, 164, 191, 98, 204, 165, 180, 117, 76, + 140, 36, 210, 172, 41, 54, 159, 8, 185, 232, 113, 196, 231, 47, 146, 120, + 51, 65, 28, 144, 254, 221, 93, 189, 194, 139, 112, 43, 71, 109, 184, 209 +}; diff --git a/contrib/preflate/preflate.h b/contrib/preflate/preflate.h new file mode 100644 index 0000000..a7caaaf --- /dev/null +++ b/contrib/preflate/preflate.h @@ -0,0 +1,21 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_H +#define PREFLATE_H + +#include "preflate_decoder.h" +#include "preflate_reencoder.h" + +#endif /* PREFLATE_H */ diff --git a/contrib/preflate/preflate_block_decoder.cpp b/contrib/preflate/preflate_block_decoder.cpp new file mode 100644 index 0000000..91bbc60 --- /dev/null +++ b/contrib/preflate/preflate_block_decoder.cpp @@ -0,0 +1,196 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "preflate_block_decoder.h" +#include "preflate_block_trees.h" +#include "support/bit_helper.h" + +PreflateBlockDecoder::PreflateBlockDecoder( + BitInputStream& input, + OutputCacheStream& output) + : _input(input) + , _output(output) + , _errorCode(OK) + , _dynamicLitLenDecoder(nullptr, 0, false, 0) + , _dynamicDistDecoder(nullptr, 0, false, 0) { +} + +bool PreflateBlockDecoder::_error(const ErrorCode code) { + _errorCode = code; + return false; +} + +bool PreflateBlockDecoder::readBlock(PreflateTokenBlock &block, bool &last) { + block.uncompressedStartPos = _output.cacheEndPos(); + int32_t earliest_reference = INT32_MAX, curPos = 0; + + if (_input.eof()) { + return false; + } + + last = _readBit() != 0; + unsigned char mode = _readBits(2); + switch (mode) { + default: + return false; + case 0: { + block.type = PreflateTokenBlock::STORED; + if (!_checkLastBitsOfByte()) { + return _error(STORED_BLOCK_PADDING_MISMATCH); + } + _skipToByte(); + size_t len = _readBits(16); + size_t ilen = _readBits(16); + if ((len ^ ilen) != 0xffff) { + return _error(STORED_BLOCK_LEN_MISMATCH); + } + block.uncompressedLen = len; + block.contextLen = 0; + return _input.copyBytesTo(_output, len) == len; + } + case 1: + case 2: + if (mode == 1) { + block.type = PreflateTokenBlock::STATIC_HUFF; + _setupStaticTables(); + } else { + block.type = PreflateTokenBlock::DYNAMIC_HUFF; + if (!_readDynamicTables(block)) { + return false; + } + } + while (true) { + if (_input.eof()) { + return false; + } + unsigned litLen = _litLenDecoder->decode(_input); + if (litLen < 256) { + _writeLiteral(litLen); + block.tokens.push_back(PreflateToken(PreflateToken::LITERAL)); + curPos++; + } else if (litLen == 256) { + block.uncompressedLen = _output.cacheEndPos() - block.uncompressedStartPos; + block.contextLen = -earliest_reference; + return true; + } else if (litLen <= PreflateConstants::L_CODES) { + unsigned lcode = litLen - PreflateConstants::LITERALS - 1; + unsigned len = PreflateConstants::MIN_MATCH + + PreflateConstants::lengthBaseTable[lcode] + + _readBits(PreflateConstants::lengthExtraTable[lcode]); + // todo: handle second version of len 258 + if (len == 258 && lcode != PreflateConstants::L_CODES - PreflateConstants::LITERALS - 2) { + return _error(BADLY_CODED_MAX_LENGTH); + } + unsigned dcode = _distDecoder->decode(_input); + if (dcode > PreflateConstants::D_CODES) { + return false; + } + unsigned dist = 1 + + PreflateConstants::distBaseTable[dcode] + + _readBits(PreflateConstants::distExtraTable[dcode]); + if (dist > _output.cacheEndPos()) { + return false; + } + _writeReference(dist, len); + block.tokens.push_back(PreflateToken(PreflateToken::REFERENCE, len, dist)); + earliest_reference = std::min(earliest_reference, curPos - (int32_t)dist); + curPos += len; + } else { + return false; + } + } + } +} + +void PreflateBlockDecoder::_setupStaticTables() { + _litLenDecoder = PreflateBlockTrees::staticLitLenTreeDecoder(); + _distDecoder = PreflateBlockTrees::staticDistTreeDecoder(); +} + +bool PreflateBlockDecoder::_readDynamicTables(PreflateTokenBlock& block) { + block.nlen = PreflateConstants::LITERALS + 1 + _readBits(5); + block.ndist = 1 + _readBits(5); + block.ncode = 4 + _readBits(4); + if (block.nlen > PreflateConstants::L_CODES || block.ndist > PreflateConstants::D_CODES) { + return false; + } + block.treecodes.clear(); + block.treecodes.reserve(block.nlen + block.ndist + block.ncode); + + unsigned char tcBitLengths[PreflateConstants::BL_CODES]; + unsigned char ldBitLengths[PreflateConstants::LD_CODES]; + memset(tcBitLengths, 0, sizeof(tcBitLengths)); + memset(ldBitLengths, 0, sizeof(ldBitLengths)); + for (unsigned i = 0, n = block.ncode; i < n; ++i) { + unsigned char tc = _readBits(3); + block.treecodes.push_back(tc); + tcBitLengths[PreflateConstants::treeCodeOrderTable[i]] = tc; + } + HuffmanDecoder tcTree(tcBitLengths, PreflateConstants::BL_CODES, true, 7); + if (tcTree.error()) { + return false; + } + for (unsigned i = 0, n = block.nlen + block.ndist; i < n; ++i) { + unsigned char code = tcTree.decode(_input); + if (code > 18) { + return false; + } + block.treecodes.push_back(code); + if (code < 16) { + ldBitLengths[i] = code; + continue; + } + unsigned char len = 0, tocopy = 0; + switch (code) { + case 16: + if (i == 0) { + return false; + } + tocopy = ldBitLengths[i - 1]; + len = 3 + _readBits(2); + break; + case 17: + tocopy = 0; + len = 3 + _readBits(3); + break; + case 18: + tocopy = 0; + len = 11 + _readBits(7); + break; + } + if (i + len > n) { + return false; + } + block.treecodes.push_back(len); + memset(ldBitLengths + i, tocopy, len); + i += len - 1; + } + if (!ldBitLengths[256]) { + return false; + } + _dynamicLitLenDecoder = HuffmanDecoder(ldBitLengths, block.nlen, true, 15); + if (_dynamicLitLenDecoder.error()) { + return false; + } + _litLenDecoder = &_dynamicLitLenDecoder; + + _dynamicDistDecoder = HuffmanDecoder(ldBitLengths + block.nlen, block.ndist, true, 15); + if (_dynamicDistDecoder.error()) { + return false; + } + _distDecoder = &_dynamicDistDecoder; + return true; +} diff --git a/contrib/preflate/preflate_block_decoder.h b/contrib/preflate/preflate_block_decoder.h new file mode 100644 index 0000000..af246a0 --- /dev/null +++ b/contrib/preflate/preflate_block_decoder.h @@ -0,0 +1,83 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_BLOCK_DECODER_H +#define PREFLATE_BLOCK_DECODER_H + +#include "preflate_constants.h" +#include "preflate_hash_chain.h" +#include "preflate_input.h" +#include "preflate_token.h" +#include "support/bitstream.h" +#include "support/huffman_decoder.h" +#include "support/outputcachestream.h" + +class PreflateBlockDecoder { +public: + enum ErrorCode { + OK, + STORED_BLOCK_LEN_MISMATCH, + STORED_BLOCK_PADDING_MISMATCH, + BADLY_CODED_MAX_LENGTH + }; + PreflateBlockDecoder(BitInputStream& input, + OutputCacheStream& output); + + bool readBlock(PreflateTokenBlock&, bool& last); + ErrorCode status() const { + return _errorCode; + } + +private: + bool _error(const ErrorCode); + + unsigned char _readBit() { + return _input.get(1); + } + unsigned _readBits(const unsigned bits) { + return _input.get(bits); + } + void _skipToByte() { + _input.skipToByte(); + } + bool _checkLastBitsOfByte() { + return _input.checkLastBitsOfByteAreZero(); + } + void _writeLiteral(const unsigned char l) { + _output.write(&l, 1); + } + void _writeReference(const size_t dist, const size_t len) { + _output.reserve(len); + if (len <= dist) { + _output.write(_output.cacheEnd() - dist, len); + } else { + const uint8_t* ptr = _output.cacheEnd() - dist; + for (size_t i = 0; i < len; ++i) { + _output.write(&ptr[i], 1); + } + } + } + void _setupStaticTables(); + bool _readDynamicTables(PreflateTokenBlock&); + + BitInputStream& _input; + OutputCacheStream& _output; + ErrorCode _errorCode; + const HuffmanDecoder* _litLenDecoder; + const HuffmanDecoder* _distDecoder; + HuffmanDecoder _dynamicLitLenDecoder; + HuffmanDecoder _dynamicDistDecoder; +}; + +#endif /* PREFLATE_BLOCK_DECODER_H */ diff --git a/contrib/preflate/preflate_block_reencoder.cpp b/contrib/preflate/preflate_block_reencoder.cpp new file mode 100644 index 0000000..a877801 --- /dev/null +++ b/contrib/preflate/preflate_block_reencoder.cpp @@ -0,0 +1,188 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "preflate_block_reencoder.h" +#include "preflate_block_trees.h" +#include "support/bit_helper.h" + +PreflateBlockReencoder::PreflateBlockReencoder( + BitOutputStream& bos, + const std::vector& uncompressedData) + : _output(bos) + , _uncompressedData(uncompressedData) + , _uncompressedDataPos(0) + , _errorCode(OK) + , _dynamicLitLenEncoder(nullptr, 0, false) + , _dynamicDistEncoder(nullptr, 0, false) { +} + +bool PreflateBlockReencoder::_error(const ErrorCode code) { + _errorCode = code; + return false; +} + +void PreflateBlockReencoder::_setupStaticTables() { + _litLenEncoder = PreflateBlockTrees::staticLitLenTreeEncoder(); + _distEncoder = PreflateBlockTrees::staticDistTreeEncoder(); +} + +bool PreflateBlockReencoder::_buildAndWriteDynamicTables(const PreflateTokenBlock& block) { + if (block.ncode < 4 || block.ncode > PreflateConstants::BL_CODES + || block.treecodes.size() < (size_t)block.ncode + || block.nlen < PreflateConstants::LITERALS + 1 + || block.nlen > PreflateConstants::L_CODES + || block.ndist < 1 || block.ndist > PreflateConstants::D_CODES) { + return _error(TREE_OUT_OF_RANGE); + } + unsigned char tcBitLengths[PreflateConstants::BL_CODES]; + unsigned char ldBitLengths[PreflateConstants::LD_CODES]; + memset(tcBitLengths, 0, sizeof(tcBitLengths)); + memset(ldBitLengths, 0, sizeof(ldBitLengths)); + + for (unsigned i = 0, n = block.ncode; i < n; ++i) { + unsigned char tc = block.treecodes[i]; + _output.put(tc, 3); + tcBitLengths[PreflateConstants::treeCodeOrderTable[i]] = tc; + } + HuffmanEncoder tcTree(tcBitLengths, PreflateConstants::BL_CODES, true); + if (tcTree.error()) { + return _error(BAD_CODE_TREE); + } + // unpack tree codes + unsigned o = 0, maxo = block.nlen + block.ndist; + for (auto i = block.treecodes.begin() + block.ncode, e = block.treecodes.end(); i != e; ++i) { + unsigned char code = *i; + if (code > 18) { + return _error(BAD_LD_TREE); + } + tcTree.encode(_output, code); + if (code < 16) { + if (o >= maxo) { + return _error(BAD_LD_TREE); + } + ldBitLengths[o++] = code; + continue; + } + if (i + 1 == e) { + return _error(BAD_LD_TREE); + } + if (code == 16 && o == 0) { + return _error(BAD_LD_TREE); + } + unsigned char len = *++i; + unsigned char tocopy = code == 16 ? ldBitLengths[o - 1] : 0; + static unsigned char repExtraBits[3] = {2, 3, 7}; + static unsigned char repOffset[3] = {3, 3, 11}; + _output.put(len - repOffset[code - 16], repExtraBits[code - 16]); + if (o + len > maxo) { + return _error(BAD_LD_TREE); + } + memset(ldBitLengths + o, tocopy, len); + o += len; + } + if (o != maxo) { + return _error(BAD_LD_TREE); + } + if (!ldBitLengths[256]) { + return _error(BAD_LD_TREE); + } + _dynamicLitLenEncoder = HuffmanEncoder(ldBitLengths, block.nlen, true); + if (_dynamicLitLenEncoder.error()) { + return _error(BAD_LD_TREE); + } + _litLenEncoder = &_dynamicLitLenEncoder; + + _dynamicDistEncoder = HuffmanEncoder(ldBitLengths + block.nlen, block.ndist, true); + if (_dynamicDistEncoder.error()) { + return _error(BAD_LD_TREE); + } + _distEncoder = &_dynamicDistEncoder; + return true; +} + +bool PreflateBlockReencoder::_writeTokens(const std::vector& tokens) { + for (size_t i = 0; i < tokens.size(); ++i) { + PreflateToken token = tokens[i]; + if (token.len == 1) { + if (_uncompressedDataPos >= _uncompressedData.size()) { + return _error(LITERAL_OUT_OF_BOUNDS); + } + unsigned char literal = _uncompressedData[_uncompressedDataPos++]; + _litLenEncoder->encode(_output, literal); + } else { + unsigned lencode = PreflateConstants::LCode(token.len); + _litLenEncoder->encode(_output, PreflateConstants::LITERALS + 1 + lencode); + unsigned lenextra = PreflateConstants::lengthExtraTable[lencode]; + if (lenextra) { + _output.put(token.len - PreflateConstants::MIN_MATCH - PreflateConstants::lengthBaseTable[lencode], lenextra); + } + unsigned distcode = PreflateConstants::DCode(token.dist); + _distEncoder->encode(_output, distcode); + unsigned distextra = PreflateConstants::distExtraTable[distcode]; + if (distextra) { + _output.put(token.dist - 1 - PreflateConstants::distBaseTable[distcode], distextra); + } + _uncompressedDataPos += token.len; + } + } + _litLenEncoder->encode(_output, PreflateConstants::LITERALS); + return true; +} + +bool PreflateBlockReencoder::writeBlock(const PreflateTokenBlock& block, bool last) { + if (status() != OK) { + return false; + } + _output.put(last, 1); // + switch (block.type) { + case PreflateTokenBlock::DYNAMIC_HUFF: + _output.put(2, 2); // + _output.put(block.nlen - PreflateConstants::LITERALS - 1, 5); + _output.put(block.ndist - 1, 5); + _output.put(block.ncode - 4, 4); + if (!_buildAndWriteDynamicTables(block)) { + return false; + } + if (!_writeTokens(block.tokens)) { + return false; + } + break; + case PreflateTokenBlock::STATIC_HUFF: + _output.put(1, 2); // + _setupStaticTables(); + if (!_writeTokens(block.tokens)) { + return false; + } + break; + case PreflateTokenBlock::STORED: + _output.put(0, 2); // + _output.fillByte(); + _output.put(block.uncompressedLen, 16); // + _output.put(~block.uncompressedLen, 16); // + if (_uncompressedDataPos + block.uncompressedLen > _uncompressedData.size()) { + return _error(LITERAL_OUT_OF_BOUNDS); + } + for (unsigned i = 0; i < block.uncompressedLen; ++i) { + unsigned char literal = _uncompressedData[_uncompressedDataPos++]; + _output.put(literal, 8); + } + break; + } + return true; +} +void PreflateBlockReencoder::flush() { + _output.flush(); +} diff --git a/contrib/preflate/preflate_block_reencoder.h b/contrib/preflate/preflate_block_reencoder.h new file mode 100644 index 0000000..e8918be --- /dev/null +++ b/contrib/preflate/preflate_block_reencoder.h @@ -0,0 +1,75 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_BLOCK_REENCODER_H +#define PREFLATE_BLOCK_REENCODER_H + +#include "preflate_constants.h" +#include "preflate_token.h" +#include "support/bitstream.h" +#include "support/huffman_encoder.h" + +class PreflateBlockReencoder { +public: + enum ErrorCode { + OK, + LITERAL_OUT_OF_BOUNDS, + TREE_OUT_OF_RANGE, + BAD_CODE_TREE, + BAD_LD_TREE, + }; + /* enum { + BUFSIZE = 1024 + }; + + std::vector output; + unsigned char buffer[BUFSIZE]; + unsigned bufferpos; + unsigned bitbuffer; + unsigned bitbuffersize;*/ + +/* unsigned short litLenDistCodeStorage[PreflateConstants::LD_CODES]; + unsigned short treeCodeStorage[PreflateConstants::BL_CODES]; + unsigned char litLenDistBitStorage[PreflateConstants::LD_CODES]; + unsigned char treeBitStorage[PreflateConstants::BL_CODES]; + const unsigned short *litLenCode, *distCode, *treeCode; + const unsigned char *litLenBits, *distBits, *treeBits;*/ + + PreflateBlockReencoder(BitOutputStream& bos, const std::vector& uncompressedData); + bool writeBlock(const PreflateTokenBlock&, const bool last); + void flush(); + + ErrorCode status() const { + return _errorCode; + } + +private: + bool _error(const ErrorCode); + + void _setupStaticTables(); + bool _buildAndWriteDynamicTables(const PreflateTokenBlock&); + bool _writeTokens(const std::vector& tokens); + + BitOutputStream& _output; + const std::vector& _uncompressedData; + size_t _uncompressedDataPos; + ErrorCode _errorCode; + + const HuffmanEncoder* _litLenEncoder; + const HuffmanEncoder* _distEncoder; + HuffmanEncoder _dynamicLitLenEncoder; + HuffmanEncoder _dynamicDistEncoder; +}; + +#endif /* PREFLATE_BLOCK_REENCODER_H */ diff --git a/contrib/preflate/preflate_block_trees.cpp b/contrib/preflate/preflate_block_trees.cpp new file mode 100644 index 0000000..6f68d60 --- /dev/null +++ b/contrib/preflate/preflate_block_trees.cpp @@ -0,0 +1,65 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "preflate_block_trees.h" +#include "support/bit_helper.h" + +static HuffmanDecoder* staticLitLenDecoder; +static HuffmanDecoder* staticDistDecoder; +static HuffmanEncoder* staticLitLenEncoder; +static HuffmanEncoder* staticDistEncoder; + +static void setLitLenBitLengths(unsigned char(&a)[288]) { + std::fill(a + 0, a + 144, 8); + std::fill(a + 144, a + 256, 9); + std::fill(a + 256, a + 280, 7); + std::fill(a + 280, a + 288, 8); +} +static void setDistBitLengths(unsigned char(&a)[32]) { + std::fill(a, a + 32, 5); +} + +const HuffmanDecoder* PreflateBlockTrees::staticLitLenTreeDecoder() { + if (!staticLitLenDecoder) { + unsigned char l_lengths[288]; + setLitLenBitLengths(l_lengths); + staticLitLenDecoder = new HuffmanDecoder(l_lengths, 288, true, 15); + } + return staticLitLenDecoder; +} +const HuffmanDecoder* PreflateBlockTrees::staticDistTreeDecoder() { + if (!staticDistDecoder) { + unsigned char d_lengths[32]; + setDistBitLengths(d_lengths); + staticDistDecoder = new HuffmanDecoder(d_lengths, 32, true, 15); + } + return staticDistDecoder; +} +const HuffmanEncoder* PreflateBlockTrees::staticLitLenTreeEncoder() { + if (!staticLitLenEncoder) { + unsigned char l_lengths[288]; + setLitLenBitLengths(l_lengths); + staticLitLenEncoder = new HuffmanEncoder(l_lengths, 288, true); + } + return staticLitLenEncoder; +} +const HuffmanEncoder* PreflateBlockTrees::staticDistTreeEncoder() { + if (!staticDistEncoder) { + unsigned char d_lengths[32]; + setDistBitLengths(d_lengths); + staticDistEncoder = new HuffmanEncoder(d_lengths, 32, true); + } + return staticDistEncoder; +} diff --git a/contrib/preflate/preflate_block_trees.h b/contrib/preflate/preflate_block_trees.h new file mode 100644 index 0000000..35cd72a --- /dev/null +++ b/contrib/preflate/preflate_block_trees.h @@ -0,0 +1,32 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_BLOCK_TREES_H +#define PREFLATE_BLOCK_TREES_H + +#include "preflate_constants.h" +#include "preflate_hash_chain.h" +#include "preflate_input.h" +#include "preflate_token.h" +#include "support/huffman_decoder.h" +#include "support/huffman_encoder.h" + +struct PreflateBlockTrees { + static const HuffmanDecoder* staticLitLenTreeDecoder(); + static const HuffmanDecoder* staticDistTreeDecoder(); + static const HuffmanEncoder* staticLitLenTreeEncoder(); + static const HuffmanEncoder* staticDistTreeEncoder(); +}; + +#endif /* PREFLATE_BLOCK_TREES_H */ diff --git a/contrib/preflate/preflate_checker.cpp b/contrib/preflate/preflate_checker.cpp new file mode 100644 index 0000000..b3259de --- /dev/null +++ b/contrib/preflate/preflate_checker.cpp @@ -0,0 +1,264 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "preflate_block_decoder.h" +#include "preflate_block_reencoder.h" +#include "preflate_checker.h" +#include "preflate_parameter_estimator.h" +#include "preflate_statistical_model.h" +#include "preflate_token_predictor.h" +#include "preflate_tree_predictor.h" +#include "preflate_unpack.h" +#include "support/bitstream.h" +#include "support/memstream.h" +#include "support/outputcachestream.h" + +#include + +bool preflate_checker(const std::vector& deflate_raw) { + printf("Checking raw deflate file of size %d\n", (int)deflate_raw.size()); + + std::vector unpacked_output; + std::vector blocks; + if (!preflate_unpack(unpacked_output, blocks, deflate_raw)) { + printf("inflating error (modified zlib)\n"); + return false; + } + printf("Unpacked data has size %d\n", (int)unpacked_output.size()); + MemStream decIn(deflate_raw); + MemStream decUnc; + BitInputStream decInBits(decIn); + OutputCacheStream decOutCache(decUnc); + std::vector blocks2; + PreflateBlockDecoder bdec(decInBits, decOutCache); + if (bdec.status() != PreflateBlockDecoder::OK) { + return false; + } + bool last; + unsigned i = 0; + do { + PreflateTokenBlock newBlock; + bool ok = bdec.readBlock(newBlock, last); + if (!ok) { + printf("inflating error (preflate)\n"); + return false; + } + if ((last && i + 1 != blocks.size()) + || (!last && i + 1 == blocks.size())) { + return false; + } + if (!isEqual(newBlock, blocks[i])) { + return false; + } + blocks2.push_back(newBlock); + ++i; + } while (!last); + decOutCache.flush(); + if (decUnc.data() != unpacked_output) { + for (unsigned i = 0, n = std::min(decUnc.data().size(), unpacked_output.size()); i < n; ++i) { + if (decUnc.data()[i] != unpacked_output[i]) { + printf("xxx %d\n", i); + } + } + return false; + } + + // Encode + PreflateParameters paramsE = estimatePreflateParameters(unpacked_output, blocks); + printf("prediction parameters: w %d, c %d, m %d, zlib %d, farL3M %d, very far M %d, M2S %d, log2CD %d\n", + paramsE.windowBits, paramsE.compLevel, paramsE.memLevel, + paramsE.zlibCompatible, paramsE.farLen3MatchesDetected, + paramsE.veryFarMatchesDetected, paramsE.matchesToStartDetected, + paramsE.log2OfMaxChainDepthM1); + + PreflateStatisticalModel modelE; + memset(&modelE, 0, sizeof(modelE)); + PreflateTokenPredictor tokenPredictorE(paramsE, unpacked_output); + PreflateTreePredictor treePredictorE(unpacked_output); + for (unsigned i = 0, n = blocks.size(); i < n; ++i) { + tokenPredictorE.analyzeBlock(i, blocks[i]); + if (tokenPredictorE.predictionFailure) { + printf("block %d: compress failed token prediction\n", i); + return false; + } + treePredictorE.analyzeBlock(i, blocks[i]); + if (treePredictorE.predictionFailure) { + printf("block %d: compress failed tree prediction\n", i); + return false; + } + tokenPredictorE.updateModel(&modelE, i); + treePredictorE.updateModel(&modelE, i); + } + + modelE.print(); + + PreflateStatisticalEncoder codecE(modelE); + codecE.encodeHeader(); + codecE.encodeParameters(paramsE); + codecE.encodeModel(); + for (unsigned i = 0, n = blocks.size(); i < n; ++i) { + tokenPredictorE.encodeBlock(&codecE, i); + if (tokenPredictorE.predictionFailure) { + printf("block %d: compress failed token encoding\n", i); + return false; + } + treePredictorE.encodeBlock(&codecE, i); + if (treePredictorE.predictionFailure) { + printf("block %d: compress failed tree encoding\n", i); + return false; + } + tokenPredictorE.encodeEOF(&codecE, i, i + 1 == blocks.size()); + } + std::vector preflate_diff = codecE.encodeFinish(); + printf("Prediction diff has size %d\n", (int)preflate_diff.size()); + + // Decode + PreflateStatisticalDecoder codecD(preflate_diff); + if (!codecD.decodeHeader()) { + printf("header decoding failed\n"); + return false; + } + PreflateParameters paramsD; + if (!codecD.decodeParameters(paramsD)) { + printf("parameter decoding failed\n"); + return false; + } + if (paramsD.strategy != paramsE.strategy) { + printf("parameter decoding failed: strategy mismatch\n"); + return false; + } + if (paramsD.huffStrategy != paramsE.huffStrategy) { + printf("parameter decoding failed: huff strategy mismatch\n"); + return false; + } + if (paramsD.windowBits != paramsE.windowBits) { + printf("parameter decoding failed: windowBits mismatch\n"); + return false; + } + if (paramsD.memLevel != paramsE.memLevel) { + printf("parameter decoding failed: memLevel mismatch\n"); + return false; + } + if (paramsD.compLevel != paramsE.compLevel) { + printf("parameter decoding failed: compLevel mismatch\n"); + return false; + } + if (paramsD.zlibCompatible != paramsE.zlibCompatible + || paramsD.farLen3MatchesDetected != paramsE.farLen3MatchesDetected + || paramsD.veryFarMatchesDetected != paramsE.veryFarMatchesDetected + || paramsD.matchesToStartDetected != paramsE.matchesToStartDetected + || paramsD.log2OfMaxChainDepthM1 != paramsE.log2OfMaxChainDepthM1) { + printf("parameter decoding failed: flag mismatch\n"); + return false; + } + + if (!codecD.decodeModel()) { + printf("model decoding failed\n"); + return false; + } + if (!isEqual(*codecD.model, *codecE.model)) { + printf("decoded model differs from original\n"); + return false; + } + + PreflateTokenPredictor tokenPredictorD(paramsD, unpacked_output); + PreflateTreePredictor treePredictorD(unpacked_output); + + MemStream mem; + BitOutputStream bos(mem); + + PreflateBlockReencoder deflater(bos, unpacked_output); + unsigned blockno = 0; + bool eof = true; + do { + if (blockno >= blocks.size()) { + printf("block number too big: org %d, new %d\n", (int)blocks.size(), blockno); + return false; + } + PreflateTokenBlock block = tokenPredictorD.decodeBlock(&codecD); + if (tokenPredictorD.predictionFailure) { + printf("block %d: token uncompress failed\n", blockno); + return false; + } + if (block.type != blocks[blockno].type) { + printf("block %d: type differs: org %d, new %d\n", blockno, blocks[blockno].type, block.type); + return false; + } + for (unsigned i = 0, n = std::min(block.tokens.size(), blocks[blockno].tokens.size()); i < n; ++i) { + PreflateToken orgToken = blocks[blockno].tokens[i]; + PreflateToken newToken = block.tokens[i]; + if (newToken.len != orgToken.len || newToken.dist != orgToken.dist) { + printf("block %d: generated token %d differs: org(%d,%d), new(%d,%d)\n", + blockno, i, orgToken.len, orgToken.dist, newToken.len, newToken.dist); + return false; + } + } + if (block.tokens.size() != blocks[blockno].tokens.size()) { + printf("block %d: differing token count: org %d, new %d\n", + blockno, (int)blocks[blockno].tokens.size(), (int)block.tokens.size()); + return false; + } + + if (!treePredictorD.decodeBlock(block, &codecD)) { + printf("block %d: tree uncompress failed\n", blockno); + return false; + } + if (treePredictorD.predictionFailure) { + printf("block %d: tree uncompress failed\n", blockno); + return false; + } + if (block.type == PreflateTokenBlock::DYNAMIC_HUFF) { + if (block.nlen != blocks[blockno].nlen) { + printf("block %d: literal/len count differs: org %d, new %d\n", + blockno, blocks[blockno].nlen, block.nlen); + return false; + } + if (block.ndist != blocks[blockno].ndist) { + printf("block %d: dist count differs: org %d, new %d\n", + blockno, blocks[blockno].ndist, block.ndist); + return false; + } + if (block.ncode != blocks[blockno].ncode) { + printf("block %d: tree code count differs: org %d, new %d\n", + blockno, blocks[blockno].ncode, block.ncode); + return false; + } + if (block.treecodes != blocks[blockno].treecodes) { + printf("block %d: generated tree codes differs\n", blockno); + return false; + } + } + eof = tokenPredictorD.decodeEOF(&codecD); + deflater.writeBlock(block, eof); + ++blockno; + } while (!eof); + deflater.flush(); + + std::vector deflate_raw_out = mem.extractData(); + for (unsigned i = 0, n = std::min(deflate_raw.size(), deflate_raw_out.size()); i < n; ++i) { + if (deflate_raw[i] != deflate_raw_out[i]) { + printf("created deflate stream differs at offset %d\n", i); + return false; + } + } + if (deflate_raw.size() != deflate_raw_out.size()) { + printf("created deflate streams differs in size: org %d, new %d\n", + (int)deflate_raw.size(), (int)deflate_raw_out.size()); + return false; + } + printf("Success\n"); + return true; +} diff --git a/contrib/preflate/preflate_checker.h b/contrib/preflate/preflate_checker.h new file mode 100644 index 0000000..f1abc9e --- /dev/null +++ b/contrib/preflate/preflate_checker.h @@ -0,0 +1,22 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_CHECKER_H +#define PREFLATE_CHECKER_H + +#include + +bool preflate_checker(const std::vector& deflate_raw); + +#endif /* PREFLATE_CHECKER_H */ diff --git a/contrib/preflate/preflate_complevel_estimator.cpp b/contrib/preflate/preflate_complevel_estimator.cpp new file mode 100644 index 0000000..fcf7910 --- /dev/null +++ b/contrib/preflate/preflate_complevel_estimator.cpp @@ -0,0 +1,195 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "preflate_complevel_estimator.h" +#include "preflate_constants.h" + +PreflateCompLevelEstimatorState::PreflateCompLevelEstimatorState( + const int wbits, + const int mbits, + const std::vector& unpacked_output_, + const std::vector& blocks_) + : predictor(slowHash, slowPreflateParserSettings[5], wbits, mbits) + , slowHash(unpacked_output_, mbits) + , fastL1Hash(unpacked_output_, mbits) + , fastL2Hash(unpacked_output_, mbits) + , fastL3Hash(unpacked_output_, mbits) + , blocks(blocks_) +{ + memset(&info, 0, sizeof(info)); + info.possibleCompressionLevels = (1 << 10) - (1 << 1); +} + +void PreflateCompLevelEstimatorState::updateHash(const unsigned len) { + if (info.possibleCompressionLevels & (1 << 1)) { + fastL1Hash.updateHash(len); + } + if (info.possibleCompressionLevels & (1 << 2)) { + fastL2Hash.updateHash(len); + } + if (info.possibleCompressionLevels & (1 << 3)) { + fastL3Hash.updateHash(len); + } + slowHash.updateHash(len); +} +void PreflateCompLevelEstimatorState::updateOrSkipSingleFastHash( + PreflateHashChainExt& hash, + const unsigned len, + const PreflateParserConfig& config) { + if (len <= config.max_lazy) { + hash.updateHash(len); + } else { + hash.skipHash(len); + } +} + +void PreflateCompLevelEstimatorState::updateOrSkipHash(const unsigned len) { + if (info.possibleCompressionLevels & (1 << 1)) { + updateOrSkipSingleFastHash(fastL1Hash, len, fastPreflateParserSettings[0]); + } + if (info.possibleCompressionLevels & (1 << 2)) { + updateOrSkipSingleFastHash(fastL2Hash, len, fastPreflateParserSettings[1]); + } + if (info.possibleCompressionLevels & (1 << 3)) { + updateOrSkipSingleFastHash(fastL3Hash, len, fastPreflateParserSettings[2]); + } + slowHash.updateHash(len); +} +bool PreflateCompLevelEstimatorState::checkMatchSingleFastHash( + const PreflateToken& token, + const PreflateHashChainExt& hash, + const PreflateParserConfig& config, + const unsigned hashHead) { + unsigned mdepth = predictor.matchDepth(hash.getHead(hashHead), token, hash); + if (mdepth > config.max_chain) { + return false; + } + return true; +} +void PreflateCompLevelEstimatorState::checkMatch(const PreflateToken& token) { + unsigned hashHead = slowHash.curHash(); + if (info.possibleCompressionLevels & (1 << 1)) { + if (!checkMatchSingleFastHash(token, fastL1Hash, fastPreflateParserSettings[0], hashHead)) { + info.possibleCompressionLevels &= ~(1 << 1); + } + } + if (info.possibleCompressionLevels & (1 << 2)) { + if (!checkMatchSingleFastHash(token, fastL2Hash, fastPreflateParserSettings[1], hashHead)) { + info.possibleCompressionLevels &= ~(1 << 2); + } + } + if (info.possibleCompressionLevels & (1 << 3)) { + if (!checkMatchSingleFastHash(token, fastL3Hash, fastPreflateParserSettings[2], hashHead)) { + info.possibleCompressionLevels &= ~(1 << 3); + } + } + + info.referenceCount++; + + unsigned short mdepth = predictor.matchDepth(slowHash.getHead(hashHead), token, slowHash); + if (mdepth >= 0x8001) { + info.unfoundReferences++; + } else { + info.maxChainDepth = std::max(info.maxChainDepth, mdepth); + } + if (token.dist == predictor.currentInputPos()) { + info.matchToStart = true; + } + if (mdepth == 0) { + info.longestDistAtHop0 = std::max(info.longestDistAtHop0, token.dist); + } else { + info.longestDistAtHop1Plus = std::max(info.longestDistAtHop1Plus, token.dist); + } + if (token.len == 3) { + info.longestLen3Dist = std::max(info.longestLen3Dist, token.dist); + } + if (info.possibleCompressionLevels & ((1 << 10) - (1 << 4))) { + for (unsigned i = 0; i < 6; ++i) { + if (!(info.possibleCompressionLevels & (1 << (4 + i)))) { + continue; + } + const PreflateParserConfig& config = slowPreflateParserSettings[i]; + if (mdepth > config.max_chain) { + info.possibleCompressionLevels &= ~(1 << (4 + i)); + } + } + } +} + +void PreflateCompLevelEstimatorState::checkDump(bool early_out) { + for (unsigned i = 0, n = blocks.size(); i < n; ++i) { + const PreflateTokenBlock& b = blocks[i]; + if (b.type == PreflateTokenBlock::STORED) { + updateHash(b.uncompressedLen); + continue; + } + for (unsigned j = 0, m = b.tokens.size(); j < m; ++j) { + const PreflateToken& t = b.tokens[j]; + if (t.len == 1) { + updateHash(1); + } else { + checkMatch(t); + updateOrSkipHash(t.len); + } + if (early_out && (info.possibleCompressionLevels & (info.possibleCompressionLevels - 1)) == 0) { + return; + } + } + } +} +void PreflateCompLevelEstimatorState::recommend() { + info.recommendedCompressionLevel = 9; + info.veryFarMatches = !(info.longestDistAtHop0 <= predictor.windowSize() - PreflateConstants::MIN_LOOKAHEAD + && info.longestDistAtHop1Plus < predictor.windowSize() - PreflateConstants::MIN_LOOKAHEAD); + info.farLen3Matches = info.longestLen3Dist > 4096; + + info.zlibCompatible = info.possibleCompressionLevels > 1 + && !info.matchToStart + && !info.veryFarMatches + && (!info.farLen3Matches || (info.possibleCompressionLevels & 0xe) != 0); + if (info.unfoundReferences) { + return; + } + + if (info.zlibCompatible && info.possibleCompressionLevels > 1) { + unsigned l = info.possibleCompressionLevels >> 1; + info.recommendedCompressionLevel = 1; + while ((l & 1) == 0) { + info.recommendedCompressionLevel++; + l >>= 1; + } + return; + } + for (int i = 0; i < 6; ++i) { + const PreflateParserConfig& config = slowPreflateParserSettings[i]; + if (info.maxChainDepth <= config.max_chain) { + info.recommendedCompressionLevel = 4 + i; + return; + } + } +} + +PreflateCompLevelInfo estimatePreflateCompLevel( + const int wbits, + const int mbits, + const std::vector& unpacked_output, + const std::vector& blocks, + const bool early_out) { + PreflateCompLevelEstimatorState state(wbits, mbits, unpacked_output, blocks); + state.checkDump(early_out); + state.recommend(); + return state.info; +} diff --git a/contrib/preflate/preflate_complevel_estimator.h b/contrib/preflate/preflate_complevel_estimator.h new file mode 100644 index 0000000..75dd3bb --- /dev/null +++ b/contrib/preflate/preflate_complevel_estimator.h @@ -0,0 +1,68 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_COMPLEVEL_ESTIMATOR_H +#define PREFLATE_COMPLEVEL_ESTIMATOR_H + +#include "preflate_predictor_state.h" +#include "preflate_token.h" + +struct PreflateCompLevelInfo { + unsigned possibleCompressionLevels; + unsigned recommendedCompressionLevel; + bool zlibCompatible; + + unsigned referenceCount; + unsigned unfoundReferences; + unsigned short maxChainDepth; + unsigned short longestLen3Dist; + unsigned short longestDistAtHop0; + unsigned short longestDistAtHop1Plus; + bool matchToStart; + bool veryFarMatches; + bool farLen3Matches; +}; + +struct PreflateCompLevelEstimatorState { + PreflatePredictorState predictor; + PreflateHashChainExt slowHash; + PreflateHashChainExt fastL1Hash; + PreflateHashChainExt fastL2Hash; + PreflateHashChainExt fastL3Hash; + const std::vector& blocks; + PreflateCompLevelInfo info; + + PreflateCompLevelEstimatorState(const int wbits, const int mbits, + const std::vector& unpacked_output, + const std::vector& blocks); + void updateHash(const unsigned len); + void updateOrSkipHash(const unsigned len); + void checkMatch(const PreflateToken& token); + void checkDump(bool early_out); + void recommend(); + +private: + void updateOrSkipSingleFastHash(PreflateHashChainExt&, const unsigned len, const PreflateParserConfig&); + bool checkMatchSingleFastHash(const PreflateToken& token, const PreflateHashChainExt&, const PreflateParserConfig&, + const unsigned hashHead); +}; + +PreflateCompLevelInfo estimatePreflateCompLevel( + const int wbits, + const int mbits, + const std::vector& unpacked_output, + const std::vector& blocks, + const bool early_out); + +#endif /* PREFLATE_COMPLEVEL_ESTIMATOR_H */ diff --git a/contrib/preflate/preflate_constants.cpp b/contrib/preflate/preflate_constants.cpp new file mode 100644 index 0000000..5488270 --- /dev/null +++ b/contrib/preflate/preflate_constants.cpp @@ -0,0 +1,87 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "preflate_block_reencoder.h" +#include "support/bit_helper.h" + +/* tables taken from zlib */ + +const unsigned char PreflateConstants::distCodeTable[512] = { + 0, 1, 2, 3, 4, 4, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 8, 8, 8, 8, + 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, + 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, + 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, + 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, + 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, + 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, + 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, + 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, + 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, + 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, + 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, + 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 0, 0, 16, 17, + 18, 18, 19, 19, 20, 20, 20, 20, 21, 21, 21, 21, 22, 22, 22, 22, 22, 22, 22, 22, + 23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, + 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, + 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, + 27, 27, 27, 27, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, + 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, + 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, + 28, 28, 28, 28, 28, 28, 28, 28, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, + 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, + 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, + 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29 +}; + +const unsigned char PreflateConstants::lengthCodeTable[MAX_MATCH - MIN_MATCH + 1 ] = { + 0, 1, 2, 3, 4, 5, 6, 7, 8, 8, 9, 9, 10, 10, 11, 11, 12, 12, 12, 12, + 13, 13, 13, 13, 14, 14, 14, 14, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, + 17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18, 18, 18, 18, 18, 19, 19, 19, 19, + 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 26, 26, 26, 26, 26, 26, 26, 26, + 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, + 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, + 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 28 +}; + +const unsigned char PreflateConstants::lengthBaseTable[LENGTH_CODES] = { + 0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 20, 24, 28, 32, 40, 48, 56, + 64, 80, 96, 112, 128, 160, 192, 224, 255 +}; + +const unsigned short PreflateConstants::distBaseTable[D_CODES] = { + 0, 1, 2, 3, 4, 6, 8, 12, 16, 24, + 32, 48, 64, 96, 128, 192, 256, 384, 512, 768, + 1024, 1536, 2048, 3072, 4096, 6144, 8192, 12288, 16384, 24576 +}; + +const unsigned char PreflateConstants::lengthExtraTable[LENGTH_CODES] = { /* extra bits for each length code */ + 0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0 +}; + +const unsigned char PreflateConstants::distExtraTable[D_CODES] = { /* extra bits for each distance code */ + 0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13 +}; + +const unsigned char PreflateConstants::treeCodeOrderTable[BL_CODES] = { + 16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15 +}; diff --git a/contrib/preflate/preflate_constants.h b/contrib/preflate/preflate_constants.h new file mode 100644 index 0000000..1a05abd --- /dev/null +++ b/contrib/preflate/preflate_constants.h @@ -0,0 +1,52 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_CONSTANTS_H +#define PREFLATE_CONSTANTS_H + +struct PreflateConstants { + enum { + LENGTH_CODES = 29, + LITERALS = 256, + L_CODES = LITERALS + 1 /* eob */ + LENGTH_CODES, + D_CODES = 30, + LD_CODES = L_CODES + D_CODES, + BL_CODES = 19, + + MIN_MATCH = 3, + MAX_MATCH = 258, + + MAX_BITS = 15, + + MIN_LOOKAHEAD = MAX_MATCH + MIN_MATCH + 1, + }; + + static const unsigned char distCodeTable[512]; + static const unsigned char lengthCodeTable[MAX_MATCH - MIN_MATCH + 1]; + static const unsigned char lengthBaseTable[LENGTH_CODES]; + static const unsigned short distBaseTable[D_CODES]; + + static const unsigned char lengthExtraTable[LENGTH_CODES]; + static const unsigned char distExtraTable[D_CODES]; + static const unsigned char treeCodeOrderTable[BL_CODES]; + + static inline unsigned DCode(const unsigned dist) { + return distCodeTable[dist <= 256 ? dist - 1 : 256 + ((dist - 1) >> 7)]; + } + static inline unsigned LCode(const unsigned len) { + return lengthCodeTable[len - MIN_MATCH]; + } +}; + +#endif /* PREFLATE_CONSTANTS_H */ diff --git a/contrib/preflate/preflate_decoder.cpp b/contrib/preflate/preflate_decoder.cpp new file mode 100644 index 0000000..aa89996 --- /dev/null +++ b/contrib/preflate/preflate_decoder.cpp @@ -0,0 +1,162 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "preflate_block_decoder.h" +#include "preflate_decoder.h" +#include "preflate_parameter_estimator.h" +#include "preflate_statistical_model.h" +#include "preflate_token_predictor.h" +#include "preflate_tree_predictor.h" +#include "preflate_unpack.h" +#include "support/bitstream.h" +#include "support/memstream.h" +#include "support/outputcachestream.h" + +bool preflate_decode(std::vector& unpacked_output, + std::vector& preflate_diff, + const std::vector& deflate_raw) { + std::vector blocks; + MemStream decIn(deflate_raw); + MemStream decUnc; + BitInputStream decInBits(decIn); + OutputCacheStream decOutCache(decUnc); + PreflateBlockDecoder bdec(decInBits, decOutCache); + if (bdec.status() != PreflateBlockDecoder::OK) { + return false; + } + bool last; + unsigned i = 0; + do { + PreflateTokenBlock newBlock; + bool ok = bdec.readBlock(newBlock, last); + if (!ok) { + return false; + } + blocks.push_back(newBlock); + ++i; + } while (!last); + decOutCache.flush(); + + unpacked_output = decUnc.extractData(); + + PreflateParameters params = estimatePreflateParameters(unpacked_output, blocks); + PreflateStatisticalModel model; + memset(&model, 0, sizeof(model)); + PreflateTokenPredictor tokenPredictor(params, unpacked_output); + PreflateTreePredictor treePredictor(unpacked_output); + for (unsigned i = 0, n = blocks.size(); i < n; ++i) { + tokenPredictor.analyzeBlock(i, blocks[i]); + treePredictor.analyzeBlock(i, blocks[i]); + if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { + return false; + } + tokenPredictor.updateModel(&model, i); + treePredictor.updateModel(&model, i); + } + PreflateStatisticalEncoder codec(model); + codec.encodeHeader(); + codec.encodeParameters(params); + codec.encodeModel(); + for (unsigned i = 0, n = blocks.size(); i < n; ++i) { + tokenPredictor.encodeBlock(&codec, i); + treePredictor.encodeBlock(&codec, i); + if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { + return false; + } + tokenPredictor.encodeEOF(&codec, i, i + 1 == blocks.size()); + } + preflate_diff = codec.encodeFinish(); + return true; +} + +bool preflate_decode(std::vector& unpacked_output, + std::vector& preflate_diff, + uint64_t& deflate_size, + InputStream& deflate_raw, + std::function block_callback) { + deflate_size = 0; + uint64_t deflate_bits = 0; + size_t prevBitPos = 0; + BitInputStream decInBits(deflate_raw); + MemStream decUnc; + OutputCacheStream decOutCache(decUnc); + PreflateBlockDecoder bdec(decInBits, decOutCache); + if (bdec.status() != PreflateBlockDecoder::OK) { + return false; + } + bool last; + unsigned i = 0; + std::vector blocks; + do { + PreflateTokenBlock newBlock; + + bool ok = bdec.readBlock(newBlock, last); + if (!ok) { + return false; + } + blocks.push_back(newBlock); + ++i; + if (decOutCache.cacheSize() >= 512 * 1024) { + decOutCache.flushUpTo(decOutCache.cacheEndPos() - (32 * 1024)); + } + deflate_bits += decInBits.bitPos() - prevBitPos; + prevBitPos = decInBits.bitPos(); + block_callback(); + } while (!last); + decOutCache.flush(); + unpacked_output = decUnc.extractData(); + deflate_size = (deflate_bits + 7) >> 3; + uint8_t remaining_bit_count = (8 - deflate_bits) & 7; + uint8_t remaining_bits = decInBits.get(remaining_bit_count); + + PreflateParameters params = estimatePreflateParameters(unpacked_output, blocks); + PreflateStatisticalModel model; + memset(&model, 0, sizeof(model)); + PreflateTokenPredictor tokenPredictor(params, unpacked_output); + PreflateTreePredictor treePredictor(unpacked_output); + for (unsigned i = 0, n = blocks.size(); i < n; ++i) { + tokenPredictor.analyzeBlock(i, blocks[i]); + treePredictor.analyzeBlock(i, blocks[i]); + if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { + return false; + } + tokenPredictor.updateModel(&model, i); + treePredictor.updateModel(&model, i); + block_callback(); + } + PreflateStatisticalEncoder codec(model); + codec.encodeHeader(); + codec.encodeParameters(params); + codec.encodeModel(); + for (unsigned i = 0, n = blocks.size(); i < n; ++i) { + tokenPredictor.encodeBlock(&codec, i); + treePredictor.encodeBlock(&codec, i); + if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { + return false; + } + tokenPredictor.encodeEOF(&codec, i, i + 1 == blocks.size()); + } + if (remaining_bit_count > 0) { + if (remaining_bits != 0) { + codec.encodeValue(1, 1); + codec.encodeValue(remaining_bits, remaining_bit_count); + } else { + codec.encodeValue(0, 1); + } + } + preflate_diff = codec.encodeFinish(); + return true; +} diff --git a/contrib/preflate/preflate_decoder.h b/contrib/preflate/preflate_decoder.h new file mode 100644 index 0000000..3358c1b --- /dev/null +++ b/contrib/preflate/preflate_decoder.h @@ -0,0 +1,33 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_DECODER_H +#define PREFLATE_DECODER_H + +#include +#include +#include "preflate_token.h" +#include "support/stream.h" + +bool preflate_decode(std::vector& unpacked_output, + std::vector& preflate_diff, + const std::vector& deflate_raw); + +bool preflate_decode(std::vector& unpacked_output, + std::vector& preflate_diff, + uint64_t& deflate_size, + InputStream& deflate_raw, + std::function block_callback); + +#endif /* PREFLATE_DECODER_H */ diff --git a/contrib/preflate/preflate_dumper.cpp b/contrib/preflate/preflate_dumper.cpp new file mode 100644 index 0000000..d9f6a60 --- /dev/null +++ b/contrib/preflate/preflate_dumper.cpp @@ -0,0 +1,145 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "zlib1.2.11.dec/zlib.h" +#include "zlib1.2.11.dec/prefdump.h" +#include "preflate_dumper.h" + +PreflateDumper::PreflateDumper() + : error(false) + , hasOpenBlock(false) + , wbits(0) + , hdr2(0) + , dictid(0) { +} + +PreflateDumper::~PreflateDumper() { +} + +void PreflateDumper::setStreamHeader(int wlen, int zhdr2) { + if (hasOpenBlock) { + error = true; + return; + } + wbits = wlen; + hdr2 = zhdr2; +} +void PreflateDumper::setDictId(unsigned id) { + if (hasOpenBlock) { + error = true; + return; + } + dictid = id; +} +void PreflateDumper::addStoredBlock(int len) { + if (hasOpenBlock) { + error = true; + return; + } + blocks.push_back(PreflateTokenBlock(PreflateTokenBlock::STORED_X, len)); +} +void PreflateDumper::addHuffBlock(bool dynamic) { + if (hasOpenBlock) { + error = true; + return; + } + openBlock.type = dynamic ? PreflateTokenBlock::DYNAMIC_HUFF : PreflateTokenBlock::STATIC_HUFF; + hasOpenBlock = true; +} +void PreflateDumper::addDynamicHuffLengths(int nlen, int ndist, int ncode) { + if (!hasOpenBlock || openBlock.type != PreflateTokenBlock::DYNAMIC_HUFF) { + error = true; + return; + } + openBlock.setHuffLengths(nlen, ndist, ncode); +} +void PreflateDumper::addDynamicHuffTreeCode(int code) { + if (!hasOpenBlock || openBlock.type != PreflateTokenBlock::DYNAMIC_HUFF) { + error = true; + return; + } + openBlock.addTreeCode(code); +} +void PreflateDumper::addLiteral() { + if (!hasOpenBlock) { + error = true; + return; + } + openBlock.tokens.push_back(PreflateToken(PreflateToken::LITERAL)); +} +void PreflateDumper::addReference(int dist, int len) { + if (!hasOpenBlock) { + error = true; + return; + } + openBlock.tokens.push_back(PreflateToken(PreflateToken::REFERENCE, len, dist)); +} +void PreflateDumper::addEOB() { + if (hasOpenBlock) { + blocks.push_back(std::move(openBlock)); + hasOpenBlock = false; + } +} + +void PreflateDumper::addUncompressedData(const unsigned char* data, const unsigned len) { + uncompressed.insert(uncompressed.end(), data, data + len); +} + +// ------------------------------------ + +void prefdump_stream_header(void* dumper, int wlen, int zhdr2) { + if (dumper) { + ((PreflateDumper*)dumper)->setStreamHeader(wlen, zhdr2); + } +} +void prefdump_dictid(void* dumper, uLong dictid) { + if (dumper) { + ((PreflateDumper*)dumper)->setDictId(dictid); + } +} +void prefdump_new_stored_block(void* dumper, int len) { + if (dumper) { + ((PreflateDumper*)dumper)->addStoredBlock(len); + } +} +void prefdump_new_huff_block(void* dumper, int dynamic) { + if (dumper) { + ((PreflateDumper*)dumper)->addHuffBlock(dynamic != 0); + } +} +void prefdump_dyn_huff_lengths(void* dumper, int nlen, int ndist, int ncode) { + if (dumper) { + ((PreflateDumper*)dumper)->addDynamicHuffLengths(nlen, ndist, ncode); + } +} +void prefdump_dyn_huff_tcode(void* dumper, int code) { + if (dumper) { + ((PreflateDumper*)dumper)->addDynamicHuffTreeCode(code); + } +} +void prefdump_literal(void* dumper) { + if (dumper) { + ((PreflateDumper*)dumper)->addLiteral(); + } +} +void prefdump_reference(void* dumper, int dist, int len) { + if (dumper) { + ((PreflateDumper*)dumper)->addReference(dist, len); + } +} +void prefdump_eob(void* dumper) { + if (dumper) { + ((PreflateDumper*)dumper)->addEOB(); + } +} diff --git a/contrib/preflate/preflate_dumper.h b/contrib/preflate/preflate_dumper.h new file mode 100644 index 0000000..9eacc6d --- /dev/null +++ b/contrib/preflate/preflate_dumper.h @@ -0,0 +1,57 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_DUMPER_H +#define PREFLATE_DUMPER_H + +#include +#include "preflate_token.h" + +class PreflateDumper { +public: + PreflateDumper(); + ~PreflateDumper(); + + void setStreamHeader(int wlen, int zhdr2); + void setDictId(unsigned id); + void addStoredBlock(int len); + void addHuffBlock(bool dynamic); + void addDynamicHuffLengths(int nlen, int ndist, int ncode); + void addDynamicHuffTreeCode(int code); + void addLiteral(); + void addReference(int dist, int len); + void addEOB(); + void addUncompressedData(const unsigned char*, const unsigned); + + const std::vector& getBlocks() const { + return blocks; + } + bool hadErrors() const { + return error; + } + const std::vector& uncompressedData() const { + return uncompressed; + } + +//private: + bool error; + bool hasOpenBlock; + PreflateTokenBlock openBlock; + unsigned char wbits, hdr2; + unsigned dictid; + std::vector blocks; + std::vector uncompressed; +}; + +#endif // PREFLATE_DUMPER_H diff --git a/contrib/preflate/preflate_hash_chain.cpp b/contrib/preflate/preflate_hash_chain.cpp new file mode 100644 index 0000000..1c92518 --- /dev/null +++ b/contrib/preflate/preflate_hash_chain.cpp @@ -0,0 +1,108 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "preflate_constants.h" +#include "preflate_hash_chain.h" + +PreflateHashChainExt::PreflateHashChainExt( + const std::vector& input_, + const unsigned char memLevel) + : _input(input_) + , totalShift(-8) { + hashBits = memLevel + 7; + hashShift = (hashBits + PreflateConstants::MIN_MATCH - 1) / PreflateConstants::MIN_MATCH; + hashMask = (1 << hashBits) - 1; + head = new unsigned short[hashMask + 1]; + prev = new unsigned short[1 << 16]; + chainDepth = new unsigned[1 << 16]; + memset(head, 0, sizeof(short) * (hashMask + 1)); + memset(prev, 0, sizeof(short) * (1 << 16)); + memset(chainDepth, 0, sizeof(unsigned) * (1 << 16)); + runningHash = 0; + if (_input.remaining() > 2) { + updateRunningHash(_input.curChar(0)); + updateRunningHash(_input.curChar(1)); + } +} +PreflateHashChainExt::~PreflateHashChainExt() { + delete[] head; + delete[] chainDepth; + delete[] prev; +} + +void PreflateHashChainExt::updateHash(const unsigned l) { + if (l > 0x180) { + unsigned l_ = l; + while (l_ > 0) { + unsigned blk = std::min(l_, 0x180u); + updateHash(blk); + l_ -= blk; + } + return; + } + + const unsigned char* b = _input.curChars(); + unsigned pos = _input.pos(); + if (pos - totalShift >= 0xfe08) { + reshift(); + } + for (unsigned i = 0; i < l; ++i) { + updateRunningHash(b[2 + i]); + unsigned h = runningHash & hashMask; + unsigned p = (pos + i) - totalShift; + chainDepth[p] = chainDepth[head[h]] + 1; + prev[p] = head[h]; + head[h] = p; + } + _input.advance(l); +} +void PreflateHashChainExt::skipHash(const unsigned l) { + const unsigned char* b = _input.curChars(); + unsigned pos = _input.pos(); + if (pos - totalShift >= 0xfe08) { + reshift(); + } + updateRunningHash(b[2]); + unsigned h = runningHash & hashMask; + unsigned p = (pos) - totalShift; + chainDepth[p] = chainDepth[head[h]] + 1; + prev[p] = head[h]; + head[h] = p; + + // Skipped data is not inserted into the hash chain, + // but we must still update the chainDepth, to avoid + // bad analysis results + // -------------------- + for (unsigned i = 1; i < l; ++i) { + unsigned p = (pos + i)-totalShift; + chainDepth[p] = 0xffff8000; + } + // l must be at least 3 + updateRunningHash(b[l]); + updateRunningHash(b[l + 1]); + _input.advance(l); +} +void PreflateHashChainExt::reshift() { + const unsigned short delta = 0x7e00; + for (unsigned i = 0, n = hashMask + 1; i < n; ++i) { + head[i] = std::max(head[i], delta) - delta; + } + for (unsigned i = delta + 8, n = 1 << 16; i < n; ++i) { + prev[i - delta] = std::max(prev[i], delta) - delta; + } + memmove(chainDepth + 8, chainDepth + 8 + delta, (0x10000 - delta) * sizeof(chainDepth[0])); + totalShift += delta; +} diff --git a/contrib/preflate/preflate_hash_chain.h b/contrib/preflate/preflate_hash_chain.h new file mode 100644 index 0000000..4e88403 --- /dev/null +++ b/contrib/preflate/preflate_hash_chain.h @@ -0,0 +1,122 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_HASH_CHAIN_H +#define PREFLATE_HASH_CHAIN_H + +#include +#include "preflate_input.h" + +struct PreflateHashIterator { + const unsigned short* chain; + const unsigned * chainDepth; + const unsigned refPos; + const unsigned maxDist; + unsigned curPos, curDist; + bool isValid; + + PreflateHashIterator( + const unsigned short* chain_, + const unsigned * depth_, + const unsigned refPos_, + const unsigned maxDist_, + unsigned startPos_) + : chain(chain_) + , chainDepth(depth_) + , refPos(refPos_) + , maxDist(maxDist_) + , curPos(startPos_) + , curDist(dist(refPos_, startPos_)) { + isValid = curDist <= maxDist; + } + + inline bool valid() const { + return isValid; + } + inline bool operator !() const { + return !isValid; + } + static inline unsigned dist(const unsigned p1, const unsigned p2) { + return p1 - p2; + } + inline unsigned dist() const { + return curDist; + } + inline unsigned depth() const { + return chainDepth[curPos]; + } + inline bool next() { + curPos = chain[curPos]; + curDist = dist(refPos, curPos); + isValid = curPos > 0 && curDist <= maxDist; + return isValid; + } +}; + +struct PreflateHashChainExt { + PreflateInput _input; + unsigned short* head; + unsigned * chainDepth; + unsigned short* prev; + unsigned char hashBits, hashShift; + unsigned short runningHash, hashMask; + unsigned totalShift; + + PreflateHashChainExt(const std::vector& input_, const unsigned char memLevel); + ~PreflateHashChainExt(); + + unsigned nextHash(const unsigned char b) const { + return ((runningHash << hashShift) ^ b); + } + unsigned nextHash(const unsigned char b1, const unsigned char b2) const { + return ((((runningHash << hashShift) ^ b1) << hashShift) ^ b2); + } + void updateRunningHash(const unsigned char b) { + runningHash = (runningHash << hashShift) ^ b; + } + void reshift(); + unsigned getHead(const unsigned hash) const { + return head[hash & hashMask]; + } + unsigned getNodeDepth(const unsigned node) const { + return chainDepth[node]; + } + + PreflateHashIterator iterateFromHead(const unsigned hash, const unsigned refPos, const unsigned maxDist) const { + return PreflateHashIterator(prev, chainDepth, refPos - totalShift, maxDist, head[hash & hashMask]); + } + PreflateHashIterator iterateFromNode(const unsigned node, const unsigned refPos, const unsigned maxDist) const { + return PreflateHashIterator(prev, chainDepth, refPos - totalShift, maxDist, node); + } + PreflateHashIterator iterateFromPos(const unsigned pos, const unsigned refPos, const unsigned maxDist) const { + return PreflateHashIterator(prev, chainDepth, refPos - totalShift, maxDist, pos - totalShift); + } + const PreflateInput& input() const { + return _input; + } + unsigned curHash() const { + return nextHash(_input.curChar(2)); + } + unsigned curPlus1Hash() const { + return nextHash(_input.curChar(2), _input.curChar(3)); + } + void updateHash(const unsigned l); + void updateHashLong(const unsigned l); + void skipHash(const unsigned l); + +private: + void _updateHashSimple(const unsigned l); +}; + +#endif /* PREFLATE_HASH_CHAIN_H */ diff --git a/contrib/preflate/preflate_info.cpp b/contrib/preflate/preflate_info.cpp new file mode 100644 index 0000000..2ed0597 --- /dev/null +++ b/contrib/preflate/preflate_info.cpp @@ -0,0 +1,55 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "preflate_info.h" + +// ----------------------------------------- + +PreflateStreamInfo extractPreflateInfo(const std::vector& blocks) { + PreflateStreamInfo result; + memset(&result, 0, sizeof(result)); + result.countBlocks = blocks.size(); + for (unsigned i = 0, n = result.countBlocks; i < n; ++i) { + const PreflateTokenBlock& b = blocks[i]; + if (b.type == PreflateTokenBlock::STORED) { + result.countStoredBlocks++; + continue; + } + if (b.type == PreflateTokenBlock::STATIC_HUFF) { + result.countStaticHuffTreeBlocks++; + } + result.tokenCount += b.tokens.size(); + result.maxTokensPerBlock = std::max(result.maxTokensPerBlock, (unsigned)b.tokens.size()); + unsigned blockMaxDist = 0; + for (unsigned j = 0, m = b.tokens.size(); j < m; ++j) { + const PreflateToken& t = b.tokens[j]; + if (t.len == 1) { + result.literalCount++; + } else { + result.referenceCount++; + blockMaxDist = std::max(blockMaxDist, (unsigned)t.dist); + } + } + result.maxDist = std::max(result.maxDist, blockMaxDist); + if (blockMaxDist == 0) { + result.countHuffBlocks++; + } else if (blockMaxDist == 1) { + result.countRLEBlocks++; + } + } + return result; +} + diff --git a/contrib/preflate/preflate_info.h b/contrib/preflate/preflate_info.h new file mode 100644 index 0000000..5e25f64 --- /dev/null +++ b/contrib/preflate/preflate_info.h @@ -0,0 +1,35 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_INFO_H +#define PREFLATE_INFO_H + +#include "preflate_token.h" + +struct PreflateStreamInfo { + unsigned tokenCount; + unsigned literalCount; + unsigned referenceCount; + unsigned maxDist; + unsigned maxTokensPerBlock; + unsigned countBlocks; + unsigned countStoredBlocks; + unsigned countHuffBlocks; + unsigned countRLEBlocks; + unsigned countStaticHuffTreeBlocks; +}; + +PreflateStreamInfo extractPreflateInfo(const std::vector& blocks); + +#endif /* PREFLATE_INFO_H */ \ No newline at end of file diff --git a/contrib/preflate/preflate_input.h b/contrib/preflate/preflate_input.h new file mode 100644 index 0000000..e239960 --- /dev/null +++ b/contrib/preflate/preflate_input.h @@ -0,0 +1,49 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_INPUT_H +#define PREFLATE_INPUT_H + +#include + +class PreflateInput { +public: + PreflateInput(const std::vector& v) + : _data(v.size() > 0 ? &v[0] : nullptr), _size(v.size()), _pos(0) {} + + const unsigned pos() const { + return _pos; + } + + const unsigned char* curChars(int offset = 0) const { + return _data + _pos + offset; + } + const unsigned char curChar(int offset = 0) const { + return _data[_pos + offset]; + } + void advance(const unsigned l) { + _pos += l; + } + const unsigned remaining() const { + return _size - _pos; + } + +private: + const unsigned char* _data; + unsigned _size; + unsigned _pos; +}; + + +#endif /* PREFLATE_INPUT_H */ diff --git a/contrib/preflate/preflate_parameter_estimator.cpp b/contrib/preflate/preflate_parameter_estimator.cpp new file mode 100644 index 0000000..8f9306a --- /dev/null +++ b/contrib/preflate/preflate_parameter_estimator.cpp @@ -0,0 +1,84 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "preflate_complevel_estimator.h" +#include "preflate_constants.h" +#include "preflate_info.h" +#include "preflate_parameter_estimator.h" +#include "preflate_token_predictor.h" +#include "support/bit_helper.h" + +unsigned char estimatePreflateMemLevel(const unsigned maxBlockSize_) { + unsigned maxBlockSize = maxBlockSize_; + unsigned mbits = 0; + while (maxBlockSize > 0) { + ++mbits; maxBlockSize >>= 1; + } + mbits = std::min(std::max(mbits, 7u), 15u); + return mbits - 6; +} + +unsigned char estimatePreflateWindowBits(const unsigned maxDist_) { + unsigned maxDist = maxDist_; + maxDist += PreflateConstants::MIN_LOOKAHEAD; + unsigned wbits = 0; + while (maxDist > 0) { + ++wbits; maxDist >>= 1; + } + wbits = std::min(std::max(wbits, 9u), 15u); + return wbits; +} + +PreflateStrategy estimatePreflateStrategy(const PreflateStreamInfo& info) { + if (info.countStoredBlocks == info.countBlocks) { + return PREFLATE_STORE; + } + if (info.countHuffBlocks == info.countBlocks) { + return PREFLATE_HUFF_ONLY; + } + if (info.countRLEBlocks == info.countBlocks) { + return PREFLATE_RLE_ONLY; + } + return PREFLATE_DEFAULT; +} + +PreflateHuffStrategy estimatePreflateHuffStrategy(const PreflateStreamInfo& info) { + if (info.countStaticHuffTreeBlocks == info.countBlocks) { + return PREFLATE_HUFF_STATIC; + } + if (info.countStaticHuffTreeBlocks == 0) { + return PREFLATE_HUFF_DYNAMIC; + } + return PREFLATE_HUFF_MIXED; +} + +PreflateParameters estimatePreflateParameters(const std::vector& unpacked_output, + const std::vector& blocks) { + PreflateStreamInfo info = extractPreflateInfo(blocks); + + PreflateParameters result; + result.windowBits = estimatePreflateWindowBits(info.maxDist); + result.memLevel = estimatePreflateMemLevel(info.maxTokensPerBlock); + result.strategy = estimatePreflateStrategy(info); + result.huffStrategy = estimatePreflateHuffStrategy(info); + PreflateCompLevelInfo cl = estimatePreflateCompLevel(result.windowBits, result.memLevel, unpacked_output, blocks, false); + result.compLevel = cl.recommendedCompressionLevel; + result.zlibCompatible = cl.zlibCompatible; + result.farLen3MatchesDetected = cl.farLen3Matches; + result.veryFarMatchesDetected = cl.veryFarMatches; + result.matchesToStartDetected = cl.matchToStart; + result.log2OfMaxChainDepthM1 = cl.maxChainDepth == 0 ? 0 : bitLength(cl.maxChainDepth - 1); + return result; +} diff --git a/contrib/preflate/preflate_parameter_estimator.h b/contrib/preflate/preflate_parameter_estimator.h new file mode 100644 index 0000000..177017d --- /dev/null +++ b/contrib/preflate/preflate_parameter_estimator.h @@ -0,0 +1,109 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_PARAMETER_ESTIMATOR_H +#define PREFLATE_PARAMETER_ESTIMATOR_H + +/* deflate has four parameters: +* - strategy: the strategy can usually be guessed by looking on the given deflate stream +* (e.g. only stored blocks -> stored, +* max distance = 0 -> huffman-only, +* max distance = 1 -> rle, +* only fixed huffman trees -> fixed-huffman-tree, +* otherwise default) +* - window bits: known by max distance, less window bits would be impossible, more window +* bits would be pointless +* - mem level: used for hash calculation and number of tokens per block +* the latter can be used to put a lower limit on mem level +* - compression level: parameters for the reference finder +* +* When reencoding a deflate stream, the predictor has to make a token proposal (either to +* encode a literal or a (dist, len) pair. A correction data stream will either accept the +* proposal, or change it to the correct values. The corrected values are then fed to the +* deflate encoder, and to the predictor. +* +* The main problem is to find the missing deflate parameters (compression level and +* mem level) to minimize the number and complexity of required corrections. +* Data streams that were encoded with zlib should get perfect recognition, +* requiring only the detected deflate parameters to be encoded for perfect reconstruction. +* Data streams from other encoders (7zip, kzip, ...) should be reconstructible with minimal +* corrective instructions, similar to reflate. +* +* kzip does not limit block size to < 64k tokens, while zlib enforces it for various reasons +* (and defaults to max 16k tokens). +* Prediction for end-of-block is therefore independent of literal/reference prediction. +* +* Mixing or interpolating the prediction from different parameter packs is +* possible, but not planned right now. +*/ + +#include "preflate_info.h" +#include "preflate_parser_config.h" +#include "preflate_token.h" + +enum PreflateStrategy { + PREFLATE_DEFAULT, + PREFLATE_RLE_ONLY, + PREFLATE_HUFF_ONLY, + PREFLATE_STORE +}; +enum PreflateHuffStrategy { + PREFLATE_HUFF_DYNAMIC, + PREFLATE_HUFF_MIXED, + PREFLATE_HUFF_STATIC, +}; + +struct PreflateParameters { + PreflateStrategy strategy; + PreflateHuffStrategy huffStrategy; + bool zlibCompatible; + unsigned char windowBits; + unsigned char memLevel; + unsigned char compLevel; + // true if matches of len 3 with a distance > 4096 are allowed + // (disallowed by zlib level 4+) + bool farLen3MatchesDetected; + // true if matches of distance >= 32768 - (MAX_MATCH + MIN_MATCH + 1) are allowed + // or > 32768 - (MAX_MATCH + MIN_MATCH + 1) if it's the first node in the hash chain + // (disallowed by zlib) + bool veryFarMatchesDetected; + // true if matches to start of stream are allowed + // (disallowed by zlib) + bool matchesToStartDetected; + // log2 of maximal found chain depth - 1 + // so, 9 to 16 have value 3 + unsigned char log2OfMaxChainDepthM1; + + + bool isFastCompressor() const { + return compLevel >= 1 && compLevel <= 3; + } + bool isSlowCompressor() const { + return compLevel >= 4 && compLevel <= 9; + } + const PreflateParserConfig& config() const { + return isFastCompressor() ? fastPreflateParserSettings[compLevel - 1] + : slowPreflateParserSettings[isSlowCompressor() ? compLevel - 4 : 5]; + } +}; + +unsigned char estimatePreflateMemLevel(const unsigned maxBlockSize); +PreflateStrategy estimatePreflateStrategy(const PreflateStreamInfo&); +PreflateHuffStrategy estimatePreflateHuffStrategy(const PreflateStreamInfo&); +unsigned char estimatePreflateWindowBits(const unsigned maxDist); + +PreflateParameters estimatePreflateParameters(const std::vector& unpacked_output, + const std::vector& blocks); + +#endif /* PREFLATE_PARAMETER_ESTIMATOR_H */ diff --git a/contrib/preflate/preflate_parser_config.cpp b/contrib/preflate/preflate_parser_config.cpp new file mode 100644 index 0000000..5356373 --- /dev/null +++ b/contrib/preflate/preflate_parser_config.cpp @@ -0,0 +1,34 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "preflate_parser_config.h" + +#include + +// ----------------------------------------- + +/* good lazy nice chain */ +const PreflateParserConfig fastPreflateParserSettings[3] = { + /* 1 */ {4, 4, 8, 4}, /* max speed, no lazy matches */ + /* 2 */ {4, 5, 16, 8}, + /* 3 */ {4, 6, 32, 32}, +}; +const PreflateParserConfig slowPreflateParserSettings[6] = { + /* 4 */ {4, 4, 16, 16}, /* lazy matches */ + /* 5 */ {8, 16, 32, 32}, + /* 6 */ {8, 16, 128, 128}, + /* 7 */ {8, 32, 128, 256}, + /* 8 */ {32, 128, 258, 1024}, + /* 9 */ {32, 258, 258, 4096}, /* max compression */ +}; diff --git a/contrib/preflate/preflate_parser_config.h b/contrib/preflate/preflate_parser_config.h new file mode 100644 index 0000000..d25d731 --- /dev/null +++ b/contrib/preflate/preflate_parser_config.h @@ -0,0 +1,35 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_PARSER_CONFIG +#define PREFLATE_PARSER_CONFIG + +/* Values for max_lazy_match, good_match and max_chain_length, depending on +* the desired pack level (0..9). The values given below have been tuned to +* exclude worst case performance for pathological files. Better values may be +* found for specific files. +*/ +struct PreflateParserConfig { + unsigned char good_length; /* reduce lazy search above this match length */ + unsigned short max_lazy; /* do not perform lazy search above this match length */ + unsigned short nice_length; /* quit search above this match length */ + unsigned short max_chain; +}; + +extern const PreflateParserConfig fastPreflateParserSettings[3]; +extern const PreflateParserConfig slowPreflateParserSettings[6]; + + +#endif +/* PREFLATE_PARSER_CONFIG */ \ No newline at end of file diff --git a/contrib/preflate/preflate_predictor_state.cpp b/contrib/preflate/preflate_predictor_state.cpp new file mode 100644 index 0000000..23a2a93 --- /dev/null +++ b/contrib/preflate/preflate_predictor_state.cpp @@ -0,0 +1,323 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "preflate_constants.h" +#include "preflate_predictor_state.h" +#include + +PreflatePredictorState::PreflatePredictorState( + const PreflateHashChainExt& hash_, + const PreflateParserConfig& config_, + const int wbits, + const int mbits) + : hash(hash_) + , windowBytes(1 << wbits) + , maxTokenCount((1 << (6 + mbits)) - 1) + , config(config_) { +} + +/* deflate has four parameters: + * - strategy: the strategy can usually be guessed by looking on the given deflate stream + * (e.g. only stored blocks -> stored, + * max distance = 0 -> huffman-only, + * max distance = 1 -> rle, + * only fixed huffman trees -> fixed-huffman-tree, + * otherwise default) + * - window bits: known by max distance, less window bits would be impossible, more window + * bits would be pointless + * - mem level: used for hash calculation and number of tokens per block + * the latter can be used to put a lower limit on mem level + * - compression level: parameters for the reference finder + * + * When reencoding a deflate stream, the predictor has to make a token proposal (either to + * encode a literal or a (dist, len) pair. A correction data stream will either accept the + * proposal, or change it to the correct values. The corrected values are then fed to the + * deflate encoder, and to the predictor. + * + * The main problem is to find the missing deflate parameters (compression level and + * mem level) to minimize the number and complexity of required corrections. + * Data streams that were encoded with zlib should get perfect recognition, + * requiring only the detected deflate parameters to be encoded for perfect reconstruction. + * Data streams from other encoders (7zip, kzip, ...) should be reconstructible with minimal + * corrective instructions, similar to reflate. + * + * kzip does not limit block size to < 64k tokens, while zlib enforces it for various reasons + * (and defaults to max 16k tokens). + * Prediction for end-of-block is therefore independent of literal/reference prediction. + * + * Mixing or interpolating the prediction from different parameter packs is + * possible, but not planned right now. + */ + +unsigned PreflatePredictorState::prefixCompare( + const unsigned char* s1, + const unsigned char* s2, + const unsigned bestLen, + const unsigned maxLen) { + if (s1[bestLen] != s2[bestLen]) { + return 0; + } + if (s1[0] != s2[0] || s1[1] != s2[1] || s1[2] != s2[2]) { + return 0; + } + + const unsigned char* scan = s2 + 2; + const unsigned char* match = s1 + 2; + const unsigned char* scanend = s2 + maxLen - 8; + +/* while (scan < scanend + && *++scan == *++match && *++scan == *++match + && *++scan == *++match && *++scan == *++match + && *++scan == *++match && *++scan == *++match + && *++scan == *++match && *++scan == *++match) { + }*/ + scanend = s2 + maxLen; + while (scan < scanend + && *++scan == *++match) { + } + + return scan - s2; +} + +PreflateToken PreflatePredictorState::match( + const unsigned hashHead, + const unsigned prevLen, + const unsigned offset, + const bool veryFarMatches, + const bool matchesToStart, + const unsigned maxDepth) { + PreflateToken bestMatch(PreflateToken::NONE); + unsigned maxLen = std::min(availableInputSize() - offset, (unsigned)PreflateConstants::MAX_MATCH); + if (maxLen < std::max(prevLen + 1, (unsigned)PreflateConstants::MIN_MATCH)) { + return bestMatch; + } + + unsigned maxDistHop0 = windowSize() - (veryFarMatches ? 0 : PreflateConstants::MIN_LOOKAHEAD); + unsigned maxDistHop1Plus = windowSize() - (veryFarMatches ? 0 : PreflateConstants::MIN_LOOKAHEAD + 1); + unsigned curPos = currentInputPos() + offset; + unsigned maxDistToStart = curPos - (matchesToStart ? 0 : 1); + unsigned curMaxDistHop0 = std::min(maxDistToStart, maxDistHop0); + unsigned curMaxDistHop1Plus = std::min(maxDistToStart, maxDistHop1Plus); + + PreflateHashIterator chainIt = iterateFromNode(hashHead, curPos, curMaxDistHop1Plus); + if (chainIt.dist() > curMaxDistHop0) { + return bestMatch; + } + + const unsigned char* input = inputCursor() + offset; + + unsigned maxChain = maxChainLength();/* max hash chain length */ + unsigned niceLen = maxDepth > 0 ? maxLen : std::min(niceMatchLength(), maxLen); + + if (prevLen >= goodMatchLength()) { + maxChain >>= 2; + } + if (maxDepth > 0) { + maxChain = maxDepth; + } + + unsigned bestLen = prevLen; + + do { + const unsigned char* match = input - chainIt.dist(); + + unsigned matchLength = prefixCompare(match, input, bestLen, maxLen); + if (matchLength > bestLen) { + bestLen = matchLength; + bestMatch = PreflateToken(PreflateToken::REFERENCE, matchLength, chainIt.dist()); + if (bestLen >= niceLen) { + break; + } + } + } while (chainIt.next() && maxChain-- > 1); + return bestMatch; +} + +unsigned short PreflatePredictorState::matchDepth( + const unsigned hashHead, + const PreflateToken& targetReference, + const PreflateHashChainExt& hash) { + unsigned curPos = currentInputPos(); + unsigned curMaxDist = std::min(curPos, windowSize()); + + unsigned startDepth = hash.getNodeDepth(hashHead); + PreflateHashIterator chainIt = hash.iterateFromPos(curPos - targetReference.dist, curPos, curMaxDist); + if (!chainIt.curPos || targetReference.dist > curMaxDist) { + return 0xffffu; + } + unsigned endDepth = chainIt.depth(); + return std::min(startDepth - endDepth, 0xffffu); +} + +PreflateNextMatchInfo PreflatePredictorState::nextMatchInfo( + const unsigned hashHead, + const PreflateToken& targetReference, + const PreflateHashChainExt& hash) { + PreflateNextMatchInfo result; + result.nextChainDepth = (unsigned short)~0u; + result.nextLen = 0; + result.nextDist = 0xffff; + unsigned maxLen = std::min(availableInputSize(), (unsigned)PreflateConstants::MAX_MATCH); + if (maxLen < (unsigned)PreflateConstants::MIN_MATCH) { + return result; + } + + unsigned maxDist = windowSize() - PreflateConstants::MIN_LOOKAHEAD - 1; + unsigned curPos = currentInputPos(); + unsigned curMaxDist = std::min(curPos - 1, maxDist); + unsigned curMaxDistAlt = std::min(curPos - 1, windowSize() - PreflateConstants::MIN_LOOKAHEAD); + + const unsigned char* input = inputCursor(); + unsigned startDepth = hash.getNodeDepth(hashHead); + unsigned maxChainOrg = maxChainLength();/* max hash chain length */ + PreflateHashIterator chainIt = hash.iterateFromPos(curPos - targetReference.dist, curPos, curMaxDist); + if (!chainIt.curPos || (hashHead == chainIt.curPos && chainIt.dist() > curMaxDistAlt) + || (hashHead != chainIt.curPos && chainIt.dist() > curMaxDist)) { + return result; + } + unsigned endDepth = chainIt.depth(); + unsigned maxChain = maxChainOrg - std::min(startDepth - endDepth, 0xffffu);/* max hash chain length */ + + unsigned bestLen = targetReference.len; + + while (maxChain > 0) { + if (!chainIt.next()) { + break; + } + const unsigned char* match = input - chainIt.dist(); + + unsigned matchLength = prefixCompare(match, input, bestLen, maxLen); + if (matchLength > bestLen) { + result.nextLen = matchLength; + result.nextChainDepth = maxChainOrg - maxChain; + result.nextDist = chainIt.dist(); + break; + } + --maxChain; + } + return result; +} + +PreflateRematchInfo PreflatePredictorState::rematchInfo( + const unsigned hashHead, + const PreflateToken& targetReference) { + PreflateRematchInfo result; + result.firstMatchDepth = 0xffff; + result.requestedMatchDepth = 0xffff; + result.condensedHops = 0; + unsigned maxLen = std::min(availableInputSize(), (unsigned)PreflateConstants::MAX_MATCH); + if (maxLen < targetReference.len) { + return result; + } + + unsigned maxDist = windowSize(); + unsigned curPos = currentInputPos(); + unsigned curMaxDist = std::min(curPos, maxDist); + + PreflateHashIterator chainIt = hash.iterateFromNode(hashHead, curPos, curMaxDist); + if (!chainIt) { + return result; + } + const unsigned char* input = inputCursor(); + + unsigned maxChainOrg = 0xffff;/* max hash chain length */ + unsigned maxChain = maxChainOrg;/* max hash chain length */ + + unsigned bestLen = targetReference.len; + + do { + const unsigned char* match = input - chainIt.dist(); + + unsigned matchLength = prefixCompare(match, input, bestLen - 1, bestLen); + if (matchLength >= bestLen) { + result.firstMatchDepth = std::min((unsigned)result.firstMatchDepth, maxChainOrg - maxChain); + result.condensedHops++; + } + if (chainIt.dist() >= targetReference.dist) { + if (chainIt.dist() == targetReference.dist) { + result.requestedMatchDepth = maxChainOrg - maxChain; + } + return result; + } + + chainIt.next(); + } while (!!chainIt && maxChain-- > 1); + return result; +} +unsigned PreflatePredictorState::firstMatch(const unsigned len) { + unsigned maxLen = std::min(availableInputSize(), (unsigned)PreflateConstants::MAX_MATCH); + if (maxLen < std::max(len, (unsigned)PreflateConstants::MIN_MATCH)) { + return 0; + } + + unsigned curPos = currentInputPos(); + unsigned curMaxDist = std::min(curPos, windowSize()); + + unsigned hash = calculateHash(); + + PreflateHashIterator chainIt = iterateFromHead(hash, curPos, curMaxDist); + if (!chainIt) { + return 0; + } + const unsigned char* input = inputCursor(); + + do { + const unsigned char* match = input - chainIt.dist(); + + unsigned matchLength = prefixCompare(match, input, len - 1, len); + if (matchLength >= len) { + return chainIt.dist(); + } + } while (chainIt.next()); + return 0; +} + +unsigned PreflatePredictorState::hopMatch(const PreflateToken& targetReference, const unsigned hops) { + if (hops == 0) { + return targetReference.dist; + } + + unsigned curPos = currentInputPos(); + unsigned errorDist = 0; + unsigned maxLen = std::min(availableInputSize(), (unsigned)PreflateConstants::MAX_MATCH); + if (maxLen < targetReference.len) { + return errorDist; + } + unsigned maxDist = windowSize(); + unsigned curMaxDist = std::min(curPos, maxDist); + + PreflateHashIterator chainIt = iterateFromDist(targetReference.dist, curPos, curMaxDist); + if (!chainIt) { + return 0; + } + + const unsigned char* input = inputCursor(); + + unsigned bestLen = targetReference.len; + for (unsigned todo = hops; todo > 0; ) { + if (!chainIt.next()) { + break; + } + + const unsigned char* match = input - chainIt.dist(); + + unsigned matchLength = prefixCompare(match, input - targetReference.dist, bestLen - 1, bestLen); + if (matchLength >= bestLen) { + if (--todo == 0) { + return chainIt.dist(); + } + } + } + return errorDist; +} diff --git a/contrib/preflate/preflate_predictor_state.h b/contrib/preflate/preflate_predictor_state.h new file mode 100644 index 0000000..67ae4c5 --- /dev/null +++ b/contrib/preflate/preflate_predictor_state.h @@ -0,0 +1,119 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_PREDICTOR_STATE_H +#define PREFLATE_PREDICTOR_STATE_H + +#include + +#include "preflate_input.h" +#include "preflate_hash_chain.h" +#include "preflate_parser_config.h" +#include "preflate_token.h" + +struct PreflatePreviousMatchInfo { + PreflateToken previousMatches[256]; +}; + +struct PreflateNextMatchInfo { + unsigned short nextChainDepth; + unsigned short nextLen; + unsigned short nextDist; +}; + +struct PreflateRematchInfo { + unsigned short firstMatchDepth; + unsigned short firstMatchDist; + unsigned short requestedMatchDepth; + unsigned short condensedHops; +}; + +struct PreflatePredictorState { + const PreflateHashChainExt& hash; + unsigned short windowBytes; + unsigned maxTokenCount; + const PreflateParserConfig& config; + + PreflatePredictorState(const PreflateHashChainExt&, + const PreflateParserConfig&, + const int wbits, + const int mbits); + + unsigned currentInputPos() const { + return hash.input().pos(); + } + const unsigned char* inputCursor() const { + return hash.input().curChars(); + } + unsigned windowSize() const { + return windowBytes; + } + unsigned availableInputSize() const { + return hash.input().remaining(); + } + unsigned maxChainLength() const { + return config.max_chain; + } + unsigned niceMatchLength() const { + return config.nice_length; + } + unsigned goodMatchLength() const { + return config.good_length; + } + unsigned lazyMatchLength() const { + return config.max_lazy; + } + unsigned calculateHash() const { + return hash.curHash(); + } + unsigned calculateHashNext() const { + return hash.curPlus1Hash(); + } + unsigned getCurrentHashHead(const unsigned hashNext) const { + return hash.getHead(hashNext); + } + + PreflateHashIterator iterateFromHead(const unsigned hash_, const unsigned refPos, const unsigned maxDist) const { + return hash.iterateFromHead(hash_, refPos, maxDist); + } + PreflateHashIterator iterateFromNode(const unsigned node_, const unsigned refPos, const unsigned maxDist) const { + return hash.iterateFromNode(node_, refPos, maxDist); + } + PreflateHashIterator iterateFromDist(const unsigned dist_, const unsigned refPos, const unsigned maxDist) const { + return hash.iterateFromPos(refPos - dist_, refPos, maxDist); + } + + static unsigned prefixCompare( + const unsigned char* s1, + const unsigned char* s2, + const unsigned bestLen, + const unsigned maxLen); + + PreflateToken match( + const unsigned hashHead, + const unsigned prevLen, + const unsigned offset, + const bool veryFarMatches, + const bool matchesToStart, + const unsigned maxDepth); + unsigned short matchDepth(const unsigned hashHead, const PreflateToken& targetReference, + const PreflateHashChainExt&); + PreflateNextMatchInfo nextMatchInfo(const unsigned hashHead, const PreflateToken& targetReference, + const PreflateHashChainExt&); + PreflateRematchInfo rematchInfo(const unsigned hashHead, const PreflateToken& targetReference); + unsigned firstMatch(const unsigned len); + unsigned hopMatch(const PreflateToken& token, const unsigned hops); +}; + +#endif /* PREFLATE_PREDICTOR_STATE_H */ diff --git a/contrib/preflate/preflate_reencoder.cpp b/contrib/preflate/preflate_reencoder.cpp new file mode 100644 index 0000000..cf555a8 --- /dev/null +++ b/contrib/preflate/preflate_reencoder.cpp @@ -0,0 +1,106 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "preflate_block_reencoder.h" +#include "preflate_reencoder.h" +#include "preflate_statistical_codec.h" +#include "preflate_token_predictor.h" +#include "preflate_tree_predictor.h" +#include "support/bitstream.h" +#include "support/memstream.h" + +bool preflate_reencode(std::vector& deflate_raw, + const std::vector& preflate_diff, + const std::vector& unpacked_input) { + PreflateStatisticalDecoder codec(preflate_diff); + if (!codec.decodeHeader()) { + return false; + } + PreflateParameters params; + if (!codec.decodeParameters(params)) { + return false; + } + if (!codec.decodeModel()) { + return false; + } + PreflateTokenPredictor tokenPredictor(params, unpacked_input); + PreflateTreePredictor treePredictor(unpacked_input); + + MemStream mem; + BitOutputStream bos(mem); + + PreflateBlockReencoder deflater(bos, unpacked_input); + bool eof = true; + do { + PreflateTokenBlock block = tokenPredictor.decodeBlock(&codec); + if (!treePredictor.decodeBlock(block, &codec)) { + return false; + } + if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { + return false; + } + eof = tokenPredictor.decodeEOF(&codec); + + deflater.writeBlock(block, eof); + } while (!eof); + deflater.flush(); + deflate_raw = mem.extractData(); + return true; +} +bool preflate_reencode(OutputStream& os, + const std::vector& preflate_diff, + const std::vector& unpacked_input, + std::function block_callback) { + PreflateStatisticalDecoder codec(preflate_diff); + if (!codec.decodeHeader()) { + return false; + } + PreflateParameters params; + if (!codec.decodeParameters(params)) { + return false; + } + if (!codec.decodeModel()) { + return false; + } + PreflateTokenPredictor tokenPredictor(params, unpacked_input); + PreflateTreePredictor treePredictor(unpacked_input); + + BitOutputStream bos(os); + + PreflateBlockReencoder deflater(bos, unpacked_input); + bool eof = true; + do { + PreflateTokenBlock block = tokenPredictor.decodeBlock(&codec); + if (!treePredictor.decodeBlock(block, &codec)) { + return false; + } + if (tokenPredictor.predictionFailure || treePredictor.predictionFailure) { + return false; + } + eof = tokenPredictor.decodeEOF(&codec); + + deflater.writeBlock(block, eof); + block_callback(); + } while (!eof); + uint8_t remaining_bit_count = (8 - bos.bitPos()) & 7; + if (remaining_bit_count > 0) { + bool non_zero_bits = codec.decodeValue(1) != 0; + if (non_zero_bits) { + bos.put(codec.decodeValue(remaining_bit_count), remaining_bit_count); + } + } + deflater.flush(); + return true; +} diff --git a/contrib/preflate/preflate_reencoder.h b/contrib/preflate/preflate_reencoder.h new file mode 100644 index 0000000..080e22d --- /dev/null +++ b/contrib/preflate/preflate_reencoder.h @@ -0,0 +1,29 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_REENCODER_H +#define PREFLATE_REENCODER_H + +#include + +bool preflate_reencode(std::vector& deflate_raw, + const std::vector& preflate_diff, + const std::vector& unpacked_input); + +bool preflate_reencode(OutputStream& os, + const std::vector& preflate_diff, + const std::vector& unpacked_input, + std::function block_callback); + +#endif /* PREFLATE_REENCODER_H */ diff --git a/contrib/preflate/preflate_statistical_codec.cpp b/contrib/preflate/preflate_statistical_codec.cpp new file mode 100644 index 0000000..5fd53b2 --- /dev/null +++ b/contrib/preflate/preflate_statistical_codec.cpp @@ -0,0 +1,789 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "preflate_parameter_estimator.h" +#include "preflate_statistical_codec.h" +#include "preflate_statistical_model.h" +#include "packARI/source/bitops.h" +#include "packARI/source/aricoder.h" +#include "support/array_helper.h" +#include "support/bit_helper.h" +#include + +template +struct PreflateCodecSubModel { + static const unsigned L = N; + PreflateCodecSubModel(const unsigned(&arr)[N]) { + memcpy(bounds, arr, sizeof(arr)); + build(); + } + PreflateCodecSubModel() { + } + void build() { + unsigned backup[N]; + for (unsigned i = 0; i < N; ++i) { + ids[i] = i; + backup[i] = bounds[i]; + } + std::sort(ids, ids + N, [=](unsigned i1, unsigned i2) { + if (backup[i1] != backup[i2]) { + return backup[i1] < backup[i2]; + } + return i1 < i2; + }); + for (unsigned i = 0; i < N; ++i) { + bounds[i] = backup[ids[i]]; + rids[ids[i]] = i; + } + unsigned sum = sumArray(bounds, N), acc, prev; + prev = bounds[0]; + bounds[0] = acc = 0; + for (unsigned i = 0; i < N; ++i) { + if (prev) { + acc += prev; + prev = bounds[i + 1]; + int diff = (((uint64_t)acc) << 16) / sum - bounds[i]; + unsigned diff_bits = bitLength(diff); + const unsigned k = 5; + if (diff > 0 && diff_bits > k) { + diff = diff & (((1 << k) - 1) << (diff_bits - k)); + } +// bounds[i + 1] = (((uint64_t)acc) << 16) / sum; + bounds[i + 1] = bounds[i] + diff; + if (bounds[i + 1] <= bounds[i]) { + bounds[i + 1] = bounds[i] + 1; + } + } else { + prev = bounds[i + 1]; + bounds[i + 1] = bounds[i]; + } + } + if (bounds[N] > 0) { + bounds[N] = 1 << 16; + } + } + void encode(aricoder* codec, const unsigned item) { + symbol s; + s.scale = 1 << 16; + s.low_count = bounds[rids[item]]; + s.high_count = bounds[rids[item] + 1]; + codec->encode(&s); + } + unsigned decode(aricoder* codec) { + symbol s; + s.scale = 1 << 16; + unsigned cnt = codec->decode_count(&s); + for (unsigned i = 0; i < N; ++i) { + if (cnt < bounds[i + 1]) { + s.low_count = bounds[i]; + s.high_count = bounds[i + 1]; + codec->decode(&s); + return ids[i]; + } + } + return 0; + } + bool isEqualTo(const PreflateCodecSubModel& m) const { + for (unsigned i = 0; i < N; ++i) { + if (bounds[i] != m.bounds[i]) { + return false; + } + if (bounds[i + 1] > 0 && ids[i] != m.ids[i]) { + return false; + } + } + if (bounds[N] != m.bounds[N]) { + return false; + } + return true; + } + + unsigned bounds[N + 1]; + unsigned short ids[N], rids[N]; +}; + +template +struct PreflateCodecCorrectionSubModel { + static const unsigned L = N; + PreflateCodecCorrectionSubModel(const unsigned(&arr1)[N + zero], + const unsigned(&arr2)[N]) + : sign() + , pos() + , neg(arr2) { + memcpy(pos.bounds, arr1 + zero, N * sizeof(unsigned)); + pos.build(); + + if (zero) { + sign.bounds[1] = arr1[0]; + } + sign.bounds[0] = sumArray(arr1 + zero, N); + sign.bounds[1 + zero] = sumArray(arr2); + sign.build(); + } + PreflateCodecCorrectionSubModel() { + } + void encode(aricoder* codec, const int value, + const unsigned refvalue, + const unsigned minvalue, + const unsigned maxvalue) { + if (value == 0) { + sign.encode(codec, 1); + return; + } + if (value > 0) { + sign.encode(codec, 0); + if (value >= (int)N) { + pos.encode(codec, N - 1); + PreflateStatisticalEncoder::encodeValue(codec, value - N, bitLength(maxvalue - N - refvalue)); + } else { + pos.encode(codec, value - 1); + } + } else { + sign.encode(codec, 1 + zero); + if (value <= -(int)N) { + neg.encode(codec, N - 1); + PreflateStatisticalEncoder::encodeValue(codec, -value - N, bitLength(refvalue - N - minvalue)); + } else { + neg.encode(codec, -value - 1); + } + } + } + int decode(aricoder* codec, + const unsigned refvalue, + const unsigned minvalue, + const unsigned maxvalue) { + int val = sign.decode(codec); + if (zero && val == 1) { + return 0; + } + if (val == 0) { + val = pos.decode(codec); + if (val >= (int)(N - 1)) { + return PreflateStatisticalDecoder::decodeValue(codec, bitLength(maxvalue - N - refvalue)) + N; + } else { + return val + 1; + } + } else { + val = neg.decode(codec); + if (val >= (int)(N - 1)) { + return -PreflateStatisticalDecoder::decodeValue(codec, bitLength(refvalue - N - minvalue)) - N; + } else { + return -val - 1; + } + } + } + bool isEqualTo(const PreflateCodecCorrectionSubModel& m) const { + return sign.isEqualTo(m.sign) + && pos.isEqualTo(m.pos) + && neg.isEqualTo(m.neg); + } + + PreflateCodecSubModel<2+zero> sign; + PreflateCodecSubModel pos; + PreflateCodecSubModel neg; +}; + +struct PreflateCodecModel { + // Blocks + PreflateCodecSubModel<3> blockType; + PreflateCodecSubModel<2> EOBMisprediction; + // Tokens + PreflateCodecSubModel<2> LITMisprediction; + PreflateCodecSubModel<2> REFMisprediction; + PreflateCodecSubModel<2> LENMisprediction; + PreflateCodecSubModel<2> DISTOnlyMisprediction; + PreflateCodecCorrectionSubModel<6,false> LENCorrection; + PreflateCodecSubModel<4> DISTAfterLenCorrection; + PreflateCodecSubModel<4> DISTOnlyCorrection; + // Trees + PreflateCodecSubModel<2> TCCountMisprediction; + PreflateCodecCorrectionSubModel<3,true> TCBitlengthCorrection; + PreflateCodecSubModel<2> LCountMisprediction; + PreflateCodecSubModel<2> DCountMisprediction; + PreflateCodecSubModel<2> LDTypeMisprediction_B; + PreflateCodecSubModel<2> LDTypeMisprediction_R; + PreflateCodecSubModel<2> LDTypeMisprediction_0s; + PreflateCodecSubModel<2> LDTypeMisprediction_0l; + PreflateCodecSubModel<2>* LDTypeMisprediction[4]; + PreflateCodecSubModel<4> LDTypeReplacement; + PreflateCodecSubModel<2> LDRepeatCountMisprediction; + PreflateCodecCorrectionSubModel<4,true> LDBitlengthCorrection; + + PreflateCodecModel(const PreflateStatisticalModel& model) + : blockType(model.blockType) + , EOBMisprediction(model.EOBMisprediction) + , LITMisprediction(model.LITMisprediction) + , REFMisprediction(model.REFMisprediction) + , LENMisprediction(model.LENMisprediction) + , DISTOnlyMisprediction(model.DISTOnlyMisprediction) + , LENCorrection(model.LENPositiveCorrection, model.LENNegativeCorrection) + , DISTAfterLenCorrection(model.DISTAfterLenCorrection) + , DISTOnlyCorrection(model.DISTOnlyCorrection) + , TCCountMisprediction(model.TCCountMisprediction) + , TCBitlengthCorrection(model.TCBitlengthPositiveCorrection, + model.TCBitlengthNegativeCorrection) + , LCountMisprediction(model.LCountMisprediction) + , DCountMisprediction(model.DCountMisprediction) + , LDTypeMisprediction_B(model.LDTypeMisprediction[0]) + , LDTypeMisprediction_R(model.LDTypeMisprediction[1]) + , LDTypeMisprediction_0s(model.LDTypeMisprediction[2]) + , LDTypeMisprediction_0l(model.LDTypeMisprediction[3]) + , LDTypeMisprediction{&LDTypeMisprediction_B, &LDTypeMisprediction_R, + &LDTypeMisprediction_0s, &LDTypeMisprediction_0l} + , LDTypeReplacement(model.LDTypeReplacement) + , LDRepeatCountMisprediction(model.LDRepeatCountMisprediction) + , LDBitlengthCorrection(model.LDBitlengthPositiveCorrection, + model.LDBitlengthNegativeCorrection) { + } + PreflateCodecModel() + : LDTypeMisprediction {&LDTypeMisprediction_B, &LDTypeMisprediction_R, + &LDTypeMisprediction_0s, &LDTypeMisprediction_0l} { + } + bool isEqualTo(const PreflateCodecModel& m) const { + return blockType.isEqualTo(m.blockType) + && EOBMisprediction.isEqualTo(m.EOBMisprediction) + && LITMisprediction.isEqualTo(m.LITMisprediction) + && REFMisprediction.isEqualTo(m.REFMisprediction) + && LENMisprediction.isEqualTo(m.LENMisprediction) + && DISTOnlyMisprediction.isEqualTo(m.DISTOnlyMisprediction) + && LENCorrection.isEqualTo(m.LENCorrection) + && DISTAfterLenCorrection.isEqualTo(m.DISTAfterLenCorrection) + && DISTOnlyCorrection.isEqualTo(m.DISTOnlyCorrection) + && TCCountMisprediction.isEqualTo(m.TCCountMisprediction) + && TCBitlengthCorrection.isEqualTo(m.TCBitlengthCorrection) + && LCountMisprediction.isEqualTo(m.LCountMisprediction) + && DCountMisprediction.isEqualTo(m.DCountMisprediction) + && LDTypeMisprediction_B.isEqualTo(m.LDTypeMisprediction_B) + && LDTypeMisprediction_R.isEqualTo(m.LDTypeMisprediction_R) + && LDTypeMisprediction_0s.isEqualTo(m.LDTypeMisprediction_0s) + && LDTypeMisprediction_0l.isEqualTo(m.LDTypeMisprediction_0l) + && LDTypeReplacement.isEqualTo(m.LDTypeReplacement) + && LDRepeatCountMisprediction.isEqualTo(m.LDRepeatCountMisprediction) + && LDBitlengthCorrection.isEqualTo(m.LDBitlengthCorrection); + } +}; + +PreflateStatisticalEncoder::PreflateStatisticalEncoder(const PreflateStatisticalModel& model_) { + data = new iostream(nullptr, TYPE_MEMORY, 0, MODE_WRITE); + codec = new aricoder(data, MODE_WRITE); + model = new PreflateCodecModel(model_); +} +PreflateStatisticalEncoder::~PreflateStatisticalEncoder() { + delete model; + delete codec; + delete data; +} + +void PreflateStatisticalEncoder::encode(const PreflateCorrectionType& type, const int value, const unsigned refvalue) { + switch (type) { + // Block + case CORR_BLOCK_TYPE: + model->blockType.encode(codec, value); + break; + case CORR_EOB_MISPREDICTION: + model->EOBMisprediction.encode(codec, value); + break; + // Tokens + case CORR_LIT_MISPREDICTION: + model->LITMisprediction.encode(codec, value); + break; + case CORR_REF_MISPREDICTION: + model->REFMisprediction.encode(codec, value); + break; + case CORR_LEN_MISPREDICTION: + model->LENMisprediction.encode(codec, value); + break; + case CORR_LEN_CORRECTION: + model->LENCorrection.encode(codec, value, refvalue, 3, 258); + break; + case CORR_DIST_AFTER_LEN_CORRECTION: + if (value >= (int)model->DISTAfterLenCorrection.L - 1) { + model->DISTAfterLenCorrection.encode(codec, model->DISTAfterLenCorrection.L - 1); + encodeValue(value - (model->DISTAfterLenCorrection.L - 1), 15); + } else { + model->DISTAfterLenCorrection.encode(codec, value); + } + break; + + case CORR_DIST_ONLY_MISPREDICTION: + model->DISTOnlyMisprediction.encode(codec, value); + break; + case CORR_DIST_ONLY_CORRECTION: + if (value >= (int)model->DISTOnlyCorrection.L - 1) { + model->DISTOnlyCorrection.encode(codec, model->DISTOnlyCorrection.L - 1); + encodeValue(value - (model->DISTOnlyCorrection.L - 1), 15); + } else { + model->DISTOnlyCorrection.encode(codec, value); + } + break; + // Trees + case CORR_L_COUNT_MISPREDICTION: + model->LCountMisprediction.encode(codec, value); + break; + case CORR_D_COUNT_MISPREDICTION: + model->DCountMisprediction.encode(codec, value); + break; + case CORR_LD_TYPE_MISPREDICTION: + model->LDTypeMisprediction[refvalue]->encode(codec, value); + break; + case CORR_LD_TYPE_REPLACEMENT: + model->LDTypeReplacement.encode(codec, value); + break; + case CORR_LD_REPEAT_MISPREDICTION: + model->LDRepeatCountMisprediction.encode(codec, value); + break; + case CORR_LD_BITLENGTH_CORRECTION: + model->LDBitlengthCorrection.encode(codec, value, refvalue, 0, 15); + break; + case CORR_TC_COUNT_MISPREDICTION: + model->TCCountMisprediction.encode(codec, value); + break; + case CORR_TC_BITLENGTH_CORRECTION: + model->TCBitlengthCorrection.encode(codec, value, refvalue, 0, 7); + break; + } +} +void PreflateStatisticalEncoder::encodeValue(aricoder* codec_, const unsigned value, const unsigned maxBits) { + symbol s; + s.scale = 1 << maxBits; + s.low_count = value; + s.high_count = value + 1; + codec_->encode(&s); +} +void PreflateStatisticalEncoder::encodeValue(const unsigned value, const unsigned maxBits) { + encodeValue(codec, value, maxBits); +} + + +struct PreflateCodecModelModel { + PreflateCodecModelModel() { + } + template + void check(const PreflateCodecSubModel& m) { + for (unsigned i = 1; i < N; ++i) { + stat_count++; + if (!m.bounds[i]) { + zero_count++; + } + } + end_stat_count++; + if (!m.bounds[N]) { + end_zero_count++; + } + } + template + void check(const PreflateCodecCorrectionSubModel& m) { + check(m.sign); + check(m.pos); + check(m.neg); + } + void check(const PreflateCodecModel& m) { + zero_count = stat_count = 0; + end_zero_count = end_stat_count = 0; + // Blocks + check(m.blockType); + check(m.EOBMisprediction); + // Tokens + check(m.LITMisprediction); + check(m.REFMisprediction); + check(m.LENMisprediction); + check(m.DISTOnlyMisprediction); + check(m.LENCorrection); + check(m.DISTAfterLenCorrection); + check(m.DISTOnlyCorrection); + // Trees + check(m.TCCountMisprediction); + check(m.TCBitlengthCorrection); + check(m.LCountMisprediction); + check(m.DCountMisprediction); + check(m.LDTypeMisprediction_B); + check(m.LDTypeMisprediction_R); + check(m.LDTypeMisprediction_0s); + check(m.LDTypeMisprediction_0l); + check(m.LDTypeReplacement); + check(m.LDRepeatCountMisprediction); + check(m.LDBitlengthCorrection); + } + + void encodeVal(aricoder* codec, const unsigned val) { + unsigned bits = bitLength(val); + // encode shift + PreflateStatisticalEncoder::encodeValue(codec, bits - 1, 4); + // and precision + if (bits >= 5) { + PreflateStatisticalEncoder::encodeValue(codec, (val >> (bits - 5)) & 0xf, 4); + } else { + PreflateStatisticalEncoder::encodeValue(codec, val & ~(1 << (bits - 1)), bits - 1); + } + } + void encodeId(aricoder* codec, + const unsigned id, const unsigned count) { + unsigned bits = bitLength(count - 1); + PreflateStatisticalEncoder::encodeValue(codec, id, bits); + } + + template + void encode(aricoder* codec, const PreflateCodecSubModel& m) { + // Mark zero values + for (unsigned i = 1; i < N + 1; ++i) { + if (!m.bounds[i]) { + codec->encode(i == N ? &ez : &z); + } else { + codec->encode(i == N ? &enz : &nz); + break; + } + } + // Transmit values + for (unsigned i = 1; i < N; ++i) { + if (m.bounds[i]) { + encodeVal(codec, m.bounds[i] - m.bounds[i - 1]); + } + } + // Transmit ids + for (unsigned i = 0; i < N; ++i) { + if (m.bounds[i + 1]) { + encodeId(codec, m.ids[i], N); + } + } + } + template + void encode(aricoder* codec, const PreflateCodecCorrectionSubModel& m) { + encode(codec, m.sign); + encode(codec, m.pos); + encode(codec, m.neg); + } + void encode(aricoder* codec, const PreflateCodecModel& m) { + PreflateStatisticalEncoder::encodeValue(codec, zero_count, bitLength(stat_count)); + PreflateStatisticalEncoder::encodeValue(codec, end_zero_count, bitLength(end_stat_count)); + + nz.scale = z.scale = stat_count; // current number of statistics + z.low_count = 0; + nz.low_count = z.high_count = zero_count; + nz.high_count = nz.scale; + + enz.scale = ez.scale = end_stat_count; // current number of statistics + ez.low_count = 0; + enz.low_count = ez.high_count = end_zero_count; + enz.high_count = end_stat_count; + + // Blocks + encode(codec, m.blockType); + encode(codec, m.EOBMisprediction); + // Tokens + encode(codec, m.LITMisprediction); + encode(codec, m.REFMisprediction); + encode(codec, m.LENMisprediction); + encode(codec, m.DISTOnlyMisprediction); + encode(codec, m.LENCorrection); + encode(codec, m.DISTAfterLenCorrection); + encode(codec, m.DISTOnlyCorrection); + // Trees + encode(codec, m.TCCountMisprediction); + encode(codec, m.TCBitlengthCorrection); + encode(codec, m.LCountMisprediction); + encode(codec, m.DCountMisprediction); + encode(codec, m.LDTypeMisprediction_B); + encode(codec, m.LDTypeMisprediction_R); + encode(codec, m.LDTypeMisprediction_0s); + encode(codec, m.LDTypeMisprediction_0l); + encode(codec, m.LDTypeReplacement); + encode(codec, m.LDRepeatCountMisprediction); + encode(codec, m.LDBitlengthCorrection); + } + + unsigned decodeVal(aricoder* codec) { + // encode shift + unsigned bits = PreflateStatisticalDecoder::decodeValue(codec, 4) + 1; + // and precision + if (bits >= 5) { + return (PreflateStatisticalDecoder::decodeValue(codec, 4) | 0x10) << (bits - 5); + } else { + return PreflateStatisticalDecoder::decodeValue(codec, bits - 1) | (1 << (bits - 1)); + } + } + unsigned decodeId(aricoder* codec, const unsigned count) { + unsigned bits = bitLength(count - 1); + return PreflateStatisticalDecoder::decodeValue(codec, bits); + } + + template + void decode(aricoder* codec, PreflateCodecSubModel& m) { + // Mark zero values + unsigned i; + m.bounds[0] = 0; + for (i = 1; i < N + 1; ++i) { + symbol* s = i == N ? &enz : &nz; + unsigned cnt = codec->decode_count(s); + if (cnt >= s->low_count) { + codec->decode(s); + break; + } + codec->decode(i == N ? &ez : &z); + m.bounds[i] = 0; + } + if (i < N + 1) { + m.bounds[N] = 1 << 16; + } + // Transmit values + unsigned j = i; + for (; i < N; ++i) { + m.bounds[i] = m.bounds[i - 1] + decodeVal(codec); + } + // Transmit ids + for (i = j - 1; i < N; ++i) { + m.ids[i] = decodeId(codec, N); + } + } + template + void decode(aricoder* codec, PreflateCodecCorrectionSubModel& m) { + decode(codec, m.sign); + decode(codec, m.pos); + decode(codec, m.neg); + } + void decode(aricoder* codec, PreflateCodecModel& m) { + zero_count = PreflateStatisticalDecoder::decodeValue(codec, bitLength(stat_count)); + end_zero_count = PreflateStatisticalDecoder::decodeValue(codec, bitLength(end_stat_count)); + + nz.scale = z.scale = stat_count; // current number of statistics + z.low_count = 0; + nz.low_count = z.high_count = zero_count; + nz.high_count = nz.scale; + + enz.scale = ez.scale = end_stat_count; // current number of statistics + ez.low_count = 0; + enz.low_count = ez.high_count = end_zero_count; + enz.high_count = end_stat_count; + + // Blocks + decode(codec, m.blockType); + decode(codec, m.EOBMisprediction); + // Tokens + decode(codec, m.LITMisprediction); + decode(codec, m.REFMisprediction); + decode(codec, m.LENMisprediction); + decode(codec, m.DISTOnlyMisprediction); + decode(codec, m.LENCorrection); + decode(codec, m.DISTAfterLenCorrection); + decode(codec, m.DISTOnlyCorrection); + // Trees + decode(codec, m.TCCountMisprediction); + decode(codec, m.TCBitlengthCorrection); + decode(codec, m.LCountMisprediction); + decode(codec, m.DCountMisprediction); + decode(codec, m.LDTypeMisprediction_B); + decode(codec, m.LDTypeMisprediction_R); + decode(codec, m.LDTypeMisprediction_0s); + decode(codec, m.LDTypeMisprediction_0l); + decode(codec, m.LDTypeReplacement); + decode(codec, m.LDRepeatCountMisprediction); + decode(codec, m.LDBitlengthCorrection); + } + + unsigned stat_count, zero_count; + unsigned end_stat_count, end_zero_count; + symbol z, nz, ez, enz; +}; + +void PreflateStatisticalEncoder::encodeModel() { + PreflateCodecModelModel modelmodel; + modelmodel.check(*model); + modelmodel.encode(codec, *model); +} + +void PreflateStatisticalEncoder::encodeHeader() { + encodeValue('P', 8); + encodeValue('F', 8); + encodeValue(0x01, 8); +} + +void PreflateStatisticalEncoder::encodeParameters(const PreflateParameters& params) { + encodeValue(params.zlibCompatible, 1); + encodeValue(params.windowBits - 8, 3); + encodeValue(params.memLevel, 4); + encodeValue(params.compLevel, 4); + encodeValue(params.strategy, 2); + encodeValue(params.huffStrategy, 2); + encodeValue(params.farLen3MatchesDetected, 1); + encodeValue(params.veryFarMatchesDetected, 1); + encodeValue(params.matchesToStartDetected, 1); + encodeValue(params.log2OfMaxChainDepthM1, 4); +} +std::vector PreflateStatisticalEncoder::encodeFinish() { + delete codec; + codec = nullptr; + std::vector result(data->getptr(), data->getptr() + data->getsize()); + delete data; + data = nullptr; +// delete model; +// model = nullptr; + return result; +} + + +PreflateStatisticalDecoder::PreflateStatisticalDecoder(const std::vector& x) : storage(x) { + data = new iostream(&storage[0], TYPE_MEMORY, storage.size(), MODE_READ); + codec = new aricoder(data, MODE_READ); + model = nullptr; +} +PreflateStatisticalDecoder::~PreflateStatisticalDecoder() { + delete model; + delete codec; + delete data; +} + +bool PreflateStatisticalDecoder::decodeHeader() { + if (decodeValue(8) != 'P') { + return false; + } + if (decodeValue(8) != 'F') { + return false; + } + if (decodeValue(8) != 0x01) { + return false; + } + return true; +} +bool PreflateStatisticalDecoder::decodeParameters(PreflateParameters& params) { + params.zlibCompatible = decodeValue(1) != 0; + params.windowBits = decodeValue(3) + 8; + params.memLevel = decodeValue(4); + params.compLevel = decodeValue(4); + + unsigned n = decodeValue(2); + switch (n) { + case PREFLATE_DEFAULT: + params.strategy = PREFLATE_DEFAULT; + break; + case PREFLATE_RLE_ONLY: + params.strategy = PREFLATE_RLE_ONLY; + break; + case PREFLATE_HUFF_ONLY: + params.strategy = PREFLATE_HUFF_ONLY; + break; + case PREFLATE_STORE: + params.strategy = PREFLATE_STORE; + break; + } + n = decodeValue(2); + switch (n) { + case PREFLATE_HUFF_DYNAMIC: + params.huffStrategy = PREFLATE_HUFF_DYNAMIC; + break; + case PREFLATE_HUFF_MIXED: + params.huffStrategy = PREFLATE_HUFF_MIXED; + break; + case PREFLATE_HUFF_STATIC: + params.huffStrategy = PREFLATE_HUFF_STATIC; + break; + } + params.farLen3MatchesDetected = decodeValue(1) != 0; + params.veryFarMatchesDetected = decodeValue(1) != 0; + params.matchesToStartDetected = decodeValue(1) != 0; + params.log2OfMaxChainDepthM1 = decodeValue(4); + return params.compLevel >= 1 && params.compLevel <= 9 + && params.memLevel >= 1 && params.memLevel <= 9 + && params.windowBits >= 8 && params.windowBits <= 15; +} +void PreflateStatisticalDecoder::decodeFinish() { + delete codec; + codec = nullptr; + delete data; + data = nullptr; + delete model; + model = nullptr; + storage.clear(); +} + +int PreflateStatisticalDecoder::decode(const PreflateCorrectionType& type, const unsigned refvalue) { + switch (type) { + // Block + case CORR_BLOCK_TYPE: + return model->blockType.decode(codec); + case CORR_EOB_MISPREDICTION: + return model->EOBMisprediction.decode(codec); + // Tokens + case CORR_LIT_MISPREDICTION: + return model->LITMisprediction.decode(codec); + case CORR_REF_MISPREDICTION: + return model->REFMisprediction.decode(codec); + case CORR_LEN_MISPREDICTION: + return model->LENMisprediction.decode(codec); + case CORR_LEN_CORRECTION: + return model->LENCorrection.decode(codec, refvalue, 3, 258); + case CORR_DIST_AFTER_LEN_CORRECTION: { + unsigned val = model->DISTAfterLenCorrection.decode(codec); + if (val >= model->DISTAfterLenCorrection.L - 1) { + return decodeValue(15) + (model->DISTAfterLenCorrection.L - 1); + } else { + return val; + } + } + case CORR_DIST_ONLY_MISPREDICTION: + return model->DISTOnlyMisprediction.decode(codec); + case CORR_DIST_ONLY_CORRECTION: { + unsigned val = model->DISTOnlyCorrection.decode(codec); + if (val >= model->DISTOnlyCorrection.L - 1) { + return decodeValue(15) + (model->DISTOnlyCorrection.L - 1); + } else { + return val; + } + } + // Trees + case CORR_L_COUNT_MISPREDICTION: + return model->LCountMisprediction.decode(codec); + case CORR_D_COUNT_MISPREDICTION: + return model->DCountMisprediction.decode(codec); + case CORR_LD_TYPE_MISPREDICTION: + return model->LDTypeMisprediction[refvalue]->decode(codec); + case CORR_LD_TYPE_REPLACEMENT: + return model->LDTypeReplacement.decode(codec); + case CORR_LD_REPEAT_MISPREDICTION: + return model->LDRepeatCountMisprediction.decode(codec); + case CORR_LD_BITLENGTH_CORRECTION: + return model->LDBitlengthCorrection.decode(codec, refvalue, 0, 15); + case CORR_TC_COUNT_MISPREDICTION: + return model->TCCountMisprediction.decode(codec); + case CORR_TC_BITLENGTH_CORRECTION: + return model->TCBitlengthCorrection.decode(codec, refvalue, 0, 7); + default: + return 0; + } +} +unsigned PreflateStatisticalDecoder::decodeValue(aricoder* codec_, const unsigned maxBits) { + symbol s; + s.scale = 1 << maxBits; + unsigned cnt = codec_->decode_count(&s); + s.low_count = cnt; + s.high_count = cnt + 1; + codec_->decode(&s); + return cnt; +} +unsigned PreflateStatisticalDecoder::decodeValue(const unsigned maxBits) { + return decodeValue(codec, maxBits); +} + +bool PreflateStatisticalDecoder::decodeModel() { + delete model; + model = new PreflateCodecModel; + PreflateCodecModelModel modelmodel; + modelmodel.check(*model); + modelmodel.decode(codec, *model); + return true; +} + +bool isEqual(const PreflateCodecModel& m1, const PreflateCodecModel& m2) { + return m1.isEqualTo(m2); +} diff --git a/contrib/preflate/preflate_statistical_codec.h b/contrib/preflate/preflate_statistical_codec.h new file mode 100644 index 0000000..5b2f656 --- /dev/null +++ b/contrib/preflate/preflate_statistical_codec.h @@ -0,0 +1,90 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_STATISTICAL_CODEC_H +#define PREFLATE_STATISTICAL_CODEC_H + +#include + +struct PreflateParameters; + +class iostream; +class aricoder; + +enum PreflateCorrectionType { + // Block + CORR_BLOCK_TYPE, + CORR_EOB_MISPREDICTION, + // Tokens + CORR_LIT_MISPREDICTION, + CORR_REF_MISPREDICTION, + CORR_LEN_MISPREDICTION, + CORR_LEN_CORRECTION, + CORR_DIST_AFTER_LEN_CORRECTION, + CORR_DIST_ONLY_MISPREDICTION, + CORR_DIST_ONLY_CORRECTION, + // Trees + CORR_L_COUNT_MISPREDICTION, + CORR_D_COUNT_MISPREDICTION, + CORR_LD_TYPE_MISPREDICTION, + CORR_LD_TYPE_REPLACEMENT, + CORR_LD_REPEAT_MISPREDICTION, + CORR_LD_BITLENGTH_CORRECTION, + CORR_TC_COUNT_MISPREDICTION, + CORR_TC_BITLENGTH_CORRECTION, +}; + +enum Codec { ENCODER, DECODER }; + +struct PreflateStatisticalModel; +struct PreflateCodecModel; + +struct PreflateStatisticalEncoder { + PreflateStatisticalEncoder(const PreflateStatisticalModel&); + ~PreflateStatisticalEncoder(); + + void encodeHeader(); + void encodeParameters(const PreflateParameters&); + void encodeModel(); + void encode(const PreflateCorrectionType& type, const int value, const unsigned refvalue = 0); + void encodeValue(const unsigned value, const unsigned maxBits); + static void encodeValue(aricoder* codec, const unsigned value, const unsigned maxBits); + std::vector encodeFinish(); + + iostream* data; + aricoder* codec; + PreflateCodecModel* model; +}; + +struct PreflateStatisticalDecoder { + PreflateStatisticalDecoder(const std::vector&); + ~PreflateStatisticalDecoder(); + + bool decodeHeader(); + bool decodeParameters(PreflateParameters&); + bool decodeModel(); + int decode(const PreflateCorrectionType& type, const unsigned refvalue = 0); + unsigned decodeValue(const unsigned maxBits); + void decodeFinish(); + static unsigned decodeValue(aricoder*, const unsigned maxBits); + + std::vector storage; + iostream* data; + aricoder* codec; + PreflateCodecModel* model; +}; + +bool isEqual(const PreflateCodecModel&, const PreflateCodecModel&); + +#endif /* PREFLATE_STATISTICAL_CODEC_H */ diff --git a/contrib/preflate/preflate_statistical_model.cpp b/contrib/preflate/preflate_statistical_model.cpp new file mode 100644 index 0000000..b82b5fa --- /dev/null +++ b/contrib/preflate/preflate_statistical_model.cpp @@ -0,0 +1,91 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "preflate_statistical_model.h" +#include "support/array_helper.h" +#include + +void printFlagStatistics(const char *txt, unsigned(&flag)[2]) { + if (flag[1]) { + printf("%s %g%% (%d)", txt, flag[1] * 100.0 / (flag[0] + flag[1]), flag[0] + flag[1]); + } +} +void printArrayStatistics(const char *txt, const char* txt2, unsigned data[], unsigned size, unsigned sum, int offset) { + bool on = false; + for (unsigned i = 0; i < size; ++i) { + if (data[i]) { + on = true; + if (i + 1 == size && txt2) { + printf("%s %g%%", txt2, data[i] * 100.0 / sum); + } else { + printf("%s%d %g%%", txt, i + offset, data[i] * 100.0 / sum); + } + } + } + if (on) { + printf(" (%d)", sum); + } +} + +void PreflateStatisticalModel::print() { + printFlagStatistics(", EOB MP", EOBMisprediction); + printFlagStatistics(", !LIT MP", LITMisprediction); + printFlagStatistics(", !REF MP", REFMisprediction); + printFlagStatistics(", !LEN MP", LENMisprediction); + printFlagStatistics(", !DIST MP", DISTOnlyMisprediction); + unsigned sum; + if (LENMisprediction) { + sum = sumArray(LENPositiveCorrection) + + sumArray(LENNegativeCorrection); + printArrayStatistics(" L+", " L+?", LENPositiveCorrection, 6, sum, 1); + printArrayStatistics(" L-", " L-?", LENNegativeCorrection, 6, sum, 1); + } + sum = sumArray(DISTAfterLenCorrection); + if (sum) { + printArrayStatistics(" L->D+", " L->D+?", DISTAfterLenCorrection, 4, sum, 0); + } + sum = sumArray(DISTOnlyCorrection); + if (sum) { + printArrayStatistics(" ->D+", " L->D+?", DISTOnlyCorrection, 4, sum, 0); + } + + printFlagStatistics(", !CT SZ MP", TCCountMisprediction); + printFlagStatistics(", !L SZ MP", LCountMisprediction); + printFlagStatistics(", !D SZ MP", DCountMisprediction); + printFlagStatistics(", !T B MP", LDTypeMisprediction[0]); + printFlagStatistics(", !T R MP", LDTypeMisprediction[1]); + printFlagStatistics(", !T 0s MP", LDTypeMisprediction[2]); + printFlagStatistics(", !T 0l MP", LDTypeMisprediction[3]); + sum = sumArray(LDTypeReplacement); + if (LDTypeReplacement[0]) { + printf(" ->T B %g", LDTypeReplacement[0] * 100.0 / sum); + } + if (LDTypeReplacement[1]) { + printf(" ->T R %g", LDTypeReplacement[1] * 100.0 / sum); + } + if (LDTypeReplacement[2]) { + printf(" ->T 0s %g", LDTypeReplacement[2] * 100.0 / sum); + } + if (LDTypeReplacement[3]) { + printf(" ->T 0l %g", LDTypeReplacement[3] * 100.0 / sum); + } + sum = sumArray(TCBitlengthPositiveCorrection) + sumArray(TCBitlengthNegativeCorrection); + printArrayStatistics(", C BL+", ", C BL+?", TCBitlengthPositiveCorrection, 4, sum, 0); + printArrayStatistics(", C BL-", ", C BL-?", TCBitlengthNegativeCorrection, 3, sum, 1); + printFlagStatistics(" LD R MP", LDRepeatCountMisprediction); + sum = sumArray(LDBitlengthPositiveCorrection) + sumArray(LDBitlengthNegativeCorrection); + printArrayStatistics(", LD BL+", ", LD BL+?", LDBitlengthPositiveCorrection, 5, sum, 0); + printArrayStatistics(", LD BL-", ", LD BL-?", LDBitlengthNegativeCorrection, 4, sum, 1); + printf("\n"); +} \ No newline at end of file diff --git a/contrib/preflate/preflate_statistical_model.h b/contrib/preflate/preflate_statistical_model.h new file mode 100644 index 0000000..5d5b873 --- /dev/null +++ b/contrib/preflate/preflate_statistical_model.h @@ -0,0 +1,52 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_STATISTICAL_MODEL_H +#define PREFLATE_STATISTICAL_MODEL_H + +//#include "packARI/source/bitops.h" +//#include "packARI/source/aricoder.h" + +struct PreflateStatisticalModel { + // Blocks + unsigned blockType[3]; // stored, dynamic huff, static huff + unsigned EOBMisprediction[2]; // no, yes + // Tokens + unsigned LITMisprediction[2]; // no, yes + unsigned REFMisprediction[2]; // no, yes + unsigned LENMisprediction[2]; // no, yes + unsigned LENPositiveCorrection[6]; // +1, +2, +3, +4, +5, +x (bytes) + unsigned LENNegativeCorrection[6]; // -1, -2, -3, -4, -5, -x (bytes) + unsigned DISTAfterLenCorrection[4]; // +0, +1, +2, +x (hops) + unsigned DISTOnlyMisprediction[2]; // no, yes + unsigned DISTOnlyCorrection[4]; // +0, +1, +2, +x (hops) + // Trees + unsigned TCCountMisprediction[2]; // no, yes + unsigned TCBitlengthPositiveCorrection[4]; // 0, +1, +2, +x + unsigned TCBitlengthNegativeCorrection[3]; // -1, -2, -x + unsigned LCountMisprediction[2]; // no, yes + unsigned DCountMisprediction[2]; // no, yes + unsigned LDTypeMisprediction[4][2]; // types: BL,REP,REPZS,REPZL; no, yes + unsigned LDTypeReplacement[4]; // types: BL,REP,REPZS,REPZL; replacement type + unsigned LDRepeatCountMisprediction[2]; // no, yes + unsigned LDBitlengthPositiveCorrection[5]; // 0, +1, +2, +3, +x + unsigned LDBitlengthNegativeCorrection[4]; // -1, -2, -3, -x, + + void print(); +}; + +// iostream data; +// aricoder ; + +#endif /* PREFLATE_STATISTICAL_MODEL_H */ diff --git a/contrib/preflate/preflate_token.cpp b/contrib/preflate/preflate_token.cpp new file mode 100644 index 0000000..f4460df --- /dev/null +++ b/contrib/preflate/preflate_token.cpp @@ -0,0 +1,43 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "preflate_token.h" + +bool isEqual(const PreflateTokenBlock& b1, const PreflateTokenBlock& b2) { + if (b1.type != b2.type) { + return false; + } +// if (b1.uncompressedLen != b2.uncompressedLen) { +// return false; +// } + if (b1.type != PreflateTokenBlock::STORED) { + if (b1.type == PreflateTokenBlock::DYNAMIC_HUFF) { + if (b1.ncode != b2.ncode || b1.nlen != b2.nlen || b1.ndist != b2.ndist) { + return false; + } + if (b1.treecodes != b2.treecodes) { + return false; + } + } + if (b1.tokens.size() != b2.tokens.size()) { + return false; + } + for (unsigned i = 0, n = b1.tokens.size(); i < n; ++i) { + if (b1.tokens[i].len != b2.tokens[i].len || b1.tokens[i].dist != b2.tokens[i].dist) { + return false; + } + } + } + return true; +} diff --git a/contrib/preflate/preflate_token.h b/contrib/preflate/preflate_token.h new file mode 100644 index 0000000..d7632be --- /dev/null +++ b/contrib/preflate/preflate_token.h @@ -0,0 +1,84 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_TOKEN_H +#define PREFLATE_TOKEN_H + +#include +#include + +/* len: 1 for literal, >= 3 for reference */ +struct PreflateToken { + enum typeLit { + LITERAL + }; + enum typeRef { + REFERENCE + }; + enum typeNon { + NONE + }; + unsigned short len; + unsigned short dist; + + PreflateToken(typeNon n) : len(0), dist(0) {} + PreflateToken(typeLit l) : len(1), dist(0) {} + PreflateToken(typeRef r, unsigned short l, unsigned short d) : len(l), dist(d) {} +}; + +struct PreflateTokenBlock { + enum Type { + STORED, DYNAMIC_HUFF, STATIC_HUFF + }; + enum StoredBlockType { + STORED_X + }; + enum HuffBlockType { + DYNAMIC_HUFF_X, STATIC_HUFF_X + }; + + Type type; + uint64_t uncompressedStartPos; + uint64_t uncompressedLen; + int32_t contextLen; // prefix size required to handle all references + unsigned short nlen, ndist, ncode; + std::vector treecodes; + std::vector tokens; + + PreflateTokenBlock() + : type(STORED) + , uncompressedLen(0) {} + PreflateTokenBlock(StoredBlockType, int len_) + : type(STORED) + , uncompressedLen(len_) {} + PreflateTokenBlock(HuffBlockType t) + : type(t == DYNAMIC_HUFF_X ? DYNAMIC_HUFF : STATIC_HUFF) + , uncompressedLen(0) {} + void setHuffLengths(int nlen_, int ndist_, int ncode_) { + nlen = nlen_; + ndist = ndist_; + ncode = ncode_; + } + void addTreeCode(int code) { + treecodes.push_back(code); + } + void addToken(const PreflateToken& token) { + tokens.push_back(token); + } +}; + +bool isEqual(const PreflateTokenBlock&, const PreflateTokenBlock&); + + +#endif /* PREFLATE_TOKEN_H */ diff --git a/contrib/preflate/preflate_token_predictor.cpp b/contrib/preflate/preflate_token_predictor.cpp new file mode 100644 index 0000000..b4cf186 --- /dev/null +++ b/contrib/preflate/preflate_token_predictor.cpp @@ -0,0 +1,447 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "preflate_constants.h" +#include "preflate_statistical_model.h" +#include "preflate_token_predictor.h" +#include "support/bit_helper.h" + +PreflateTokenPredictor::PreflateTokenPredictor( + const PreflateParameters& params_, + const std::vector& dump) + : state(hash, params_.config(), params_.windowBits, params_.memLevel) + , hash(dump, params_.memLevel) + , params(params_) + , predictionFailure(false) + , fast(params_.isFastCompressor()) + , prevLen(0) + , pendingToken(PreflateToken::NONE) + , emptyBlockAtEnd(false) { + + if (state.availableInputSize() >= 2) { + hash.updateRunningHash(state.inputCursor()[0]); + hash.updateRunningHash(state.inputCursor()[1]); + } +} + +bool PreflateTokenPredictor::predictEOB() { + return state.availableInputSize() == 0 || currentTokenCount == state.maxTokenCount; +} +void PreflateTokenPredictor::commitToken(const PreflateToken& t) { + if (fast && t.len > state.lazyMatchLength()) { + hash.skipHash(t.len); + } else { + hash.updateHash(t.len); + } +} +# define TOO_FAR 4096 +/* Matches of length 3 are discarded if their distance exceeds TOO_FAR */ + +PreflateToken PreflateTokenPredictor::predictToken() { + if (state.currentInputPos() == 0 || state.availableInputSize() < PreflateConstants::MIN_MATCH) { + return PreflateToken(PreflateToken::LITERAL); + } + unsigned hash = state.calculateHash(); + unsigned head = state.getCurrentHashHead(hash); + PreflateToken match(PreflateToken::NONE); + + if (pendingToken.len > 1) { + match = pendingToken; + } else { + match = state.match(head, prevLen, 0, + params.veryFarMatchesDetected, + params.matchesToStartDetected, + params.zlibCompatible ? 0 : (1 << params.log2OfMaxChainDepthM1)); + } + prevLen = 0; + pendingToken = PreflateToken(PreflateToken::NONE); + if (match.len < PreflateConstants::MIN_MATCH) { + return PreflateToken(PreflateToken::LITERAL); + } + if (fast) { + return match; + } + if (match.len == 3 && match.dist > TOO_FAR) { + return PreflateToken(PreflateToken::LITERAL); + } + + if (match.len < state.lazyMatchLength() && state.availableInputSize() >= (unsigned)match.len + 2) { + unsigned hashNext = state.calculateHashNext(); + unsigned headNext = state.getCurrentHashHead(hashNext); + PreflateToken matchNext = state.match(headNext, match.len, 1, + params.veryFarMatchesDetected, + params.matchesToStartDetected, + params.zlibCompatible ? 0 : (2 << params.log2OfMaxChainDepthM1)); + if (((hashNext ^ hash) & this->hash.hashMask) == 0) { + unsigned maxSize = std::min(state.availableInputSize() - 1, (unsigned)PreflateConstants::MAX_MATCH); + unsigned rle = 1; + const unsigned char *c = state.inputCursor(); + unsigned char b = c[0]; + while (rle < maxSize && c[1 + rle] == b) { + ++rle; + } + if (rle >= matchNext.len) { + matchNext.len = rle; + matchNext.dist = 1; + } + } + if (matchNext.len > match.len) { + prevLen = match.len; + pendingToken = matchNext; + if (!params.zlibCompatible) { + prevLen = 0; + pendingToken = PreflateToken(PreflateToken::NONE); + } + return PreflateToken(PreflateToken::LITERAL); + } + } + + return match; +} +bool PreflateTokenPredictor::repredictReference(PreflateToken& token) { + if (state.currentInputPos() == 0 || state.availableInputSize() < PreflateConstants::MIN_MATCH) { + return false; + } + unsigned hash = state.calculateHash(); + unsigned head = state.getCurrentHashHead(hash); + PreflateToken match = state.match(head, /*prevLen*/0, 0, + params.veryFarMatchesDetected, + params.matchesToStartDetected, + (2 << params.log2OfMaxChainDepthM1)); + prevLen = 0; + pendingToken = PreflateToken(PreflateToken::NONE); + if (match.len < PreflateConstants::MIN_MATCH) { + return false; + } + token = match; + return true; +} +PreflateRematchInfo PreflateTokenPredictor::repredictMatch(const PreflateToken& token) { + unsigned hash = state.calculateHash(); + unsigned head = state.getCurrentHashHead(hash); + PreflateRematchInfo i = state.rematchInfo(head, token); + return i; +} +unsigned PreflateTokenPredictor::recalculateDistance(const PreflateToken& token, const unsigned hops) { + return state.hopMatch(token, hops); +} + +void PreflateTokenPredictor::analyzeBlock( + const unsigned blockno, + const PreflateTokenBlock& block) { + currentTokenCount = 0; + prevLen = 0; + pendingToken = PreflateToken(PreflateToken::NONE); + if (blockno != analysisResults.size() || predictionFailure) { + return; + } + analysisResults.push_back(BlockAnalysisResult()); + BlockAnalysisResult& analysis = analysisResults[blockno]; + + analysis.type = block.type; + analysis.tokenCount = block.tokens.size(); + analysis.tokenInfo.resize(analysis.tokenCount); + analysis.blockSizePredicted = true; + analysis.inputEOF = false; + + if (analysis.type == PreflateTokenBlock::STORED) { + analysis.tokenCount = block.uncompressedLen; + hash.updateHash(block.uncompressedLen); + analysis.inputEOF = state.availableInputSize() == 0; + return; + } + + for (unsigned i = 0, n = block.tokens.size(); i < n; ++i) { + PreflateToken targetToken = block.tokens[i]; + if (targetToken.len > 258) { + predictionFailure = true; + return; + } + if (predictEOB()) { + analysis.blockSizePredicted = false; + } + PreflateToken predictedToken = predictToken(); + + if (targetToken.len == 1) { + if (predictedToken.len > 1) { + analysis.tokenInfo[currentTokenCount] = 2; // badly predicted LIT + } else { + analysis.tokenInfo[currentTokenCount] = 0; // perfectly predicted LIT + } + } else { + if (predictedToken.len == 1) { + analysis.tokenInfo[currentTokenCount] = 3; // badly predicted REF + if (!repredictReference(predictedToken)) { + predictionFailure = true; + return; + } + } else { + analysis.tokenInfo[currentTokenCount] = 1; // well predicted REF + } + PreflateRematchInfo rematch; + if (predictedToken.len != targetToken.len) { + analysis.tokenInfo[currentTokenCount] += 4; // bad LEN prediction, adds two corrective actions + analysis.correctives.push_back(predictedToken.len); + analysis.correctives.push_back(targetToken.len - predictedToken.len); + rematch = repredictMatch(targetToken); + + if (rematch.requestedMatchDepth >= 0xffff) { + predictionFailure = true; + return; + } + analysis.correctives.push_back(rematch.condensedHops - 1); + } else { + if (targetToken.dist != predictedToken.dist) { + analysis.tokenInfo[currentTokenCount] += 8; // bad DIST ONLY prediction, adds one corrective action + rematch = repredictMatch(targetToken); + + if (rematch.requestedMatchDepth >= 0xffff) { + predictionFailure = true; + return; + } + analysis.correctives.push_back(rematch.condensedHops - 1); + } + } + } + commitToken(targetToken); + ++currentTokenCount; + } + if (!predictEOB()) { + analysis.blockSizePredicted = false; + } + analysis.inputEOF = state.availableInputSize() == 0; +} + +void PreflateTokenPredictor::encodeBlock( + PreflateStatisticalEncoder* codec, + const unsigned blockno) { + BlockAnalysisResult& analysis = analysisResults[blockno]; + + codec->encode(CORR_BLOCK_TYPE, analysis.type); + + if (analysis.type == PreflateTokenBlock::STORED) { + codec->encodeValue(analysis.tokenCount, 16); + return; + } + + codec->encode(CORR_EOB_MISPREDICTION, !analysis.blockSizePredicted); + if (!analysis.blockSizePredicted) { + unsigned blocksizeBits = bitLength(analysis.tokenCount); + codec->encodeValue(blocksizeBits, 5); + if (blocksizeBits >= 2) { + codec->encodeValue(analysis.tokenCount, blocksizeBits); + } + } + + unsigned correctivePos = 0; + for (unsigned i = 0, n = analysis.tokenCount; i < n; ++i) { + unsigned char info = analysis.tokenInfo[i]; + switch (info & 3) { + case 0: // well predicted LIT + codec->encode(CORR_LIT_MISPREDICTION, 0); + continue; + case 2: // badly predicted LIT + codec->encode(CORR_REF_MISPREDICTION, 1); + continue; + case 1: // well predicted REF + codec->encode(CORR_REF_MISPREDICTION, 0); + break; + case 3: // badly predicted REF + codec->encode(CORR_LIT_MISPREDICTION, 1); + break; + } + codec->encode(CORR_LEN_MISPREDICTION, (info & 4) != 0); + if (info & 4) { + int pred = analysis.correctives[correctivePos++]; + int diff = analysis.correctives[correctivePos++]; + int hops = analysis.correctives[correctivePos++]; + codec->encode(CORR_LEN_CORRECTION, diff, pred); + codec->encode(CORR_DIST_AFTER_LEN_CORRECTION, hops); + } else { + codec->encode(CORR_DIST_ONLY_MISPREDICTION, (info & 8) != 0); + if (info & 8) { + int hops = analysis.correctives[correctivePos++]; + codec->encode(CORR_DIST_ONLY_CORRECTION, hops); + } + } + } +} +void PreflateTokenPredictor::encodeEOF( + PreflateStatisticalEncoder* codec, + const unsigned blockno, + const bool lastBlock) { + BlockAnalysisResult& analysis = analysisResults[blockno]; + + if (analysis.inputEOF) { + codec->encodeValue(!lastBlock, 1); + } else { + // If we still have input left, this shouldn't be the last block + if (lastBlock) { + predictionFailure = true; + } + } +} + +void PreflateTokenPredictor::updateModel( + PreflateStatisticalModel* model, + const unsigned blockno) { + BlockAnalysisResult& analysis = analysisResults[blockno]; + + model->blockType[analysis.type]++; + + if (analysis.type == PreflateTokenBlock::STORED) { + return; + } + + model->EOBMisprediction[!analysis.blockSizePredicted]++; + + unsigned correctivePos = 0; + for (unsigned i = 0, n = analysis.tokenCount; i < n; ++i) { + unsigned char info = analysis.tokenInfo[i]; + switch (info & 3) { + case 0: // well predicted LIT + model->LITMisprediction[0]++; + continue; + case 2: // badly predicted LIT + model->REFMisprediction[1]++; + continue; + case 1: // well predicted REF + model->REFMisprediction[0]++; + break; + case 3: // badly predicted REF + model->LITMisprediction[1]++; + break; + } + model->LENMisprediction[(info & 4) != 0]++; + if (info & 4) { + /*int pred = analysis.correctives[*/correctivePos++/*]*/; + int diff = analysis.correctives[correctivePos++]; + int hops = analysis.correctives[correctivePos++]; + if (diff > 0) { + model->LENPositiveCorrection[std::min(5, diff - 1)]++; + } else { + model->LENNegativeCorrection[std::min(5, -diff - 1)]++; + } + model->DISTAfterLenCorrection[std::min(3, hops)]++; + } else { + model->DISTOnlyMisprediction[(info & 8) != 0]++; + if (info & 8) { + int hops = analysis.correctives[correctivePos++]; + model->DISTOnlyCorrection[std::min(3, hops)]++; + } + } + } +} + +PreflateTokenBlock PreflateTokenPredictor::decodeBlock( + PreflateStatisticalDecoder* codec) { + PreflateTokenBlock block; + currentTokenCount = 0; + prevLen = 0; + pendingToken = PreflateToken(PreflateToken::NONE); + unsigned blocksize = 0; + bool checkEOB = true; + unsigned bt = codec->decode(CORR_BLOCK_TYPE); + switch (bt) { + case PreflateTokenBlock::STORED: + block.type = PreflateTokenBlock::STORED; + block.uncompressedLen = codec->decodeValue(16); + hash.updateHash(block.uncompressedLen); + return block; + case PreflateTokenBlock::STATIC_HUFF: + block.type = PreflateTokenBlock::STATIC_HUFF; + break; + case PreflateTokenBlock::DYNAMIC_HUFF: + block.type = PreflateTokenBlock::DYNAMIC_HUFF; + break; + } + + if (codec->decode(CORR_EOB_MISPREDICTION)) { + unsigned blocksizeBits = codec->decodeValue(5); + if (blocksizeBits >= 2) { + blocksize = codec->decodeValue(blocksizeBits); + } else { + blocksize = blocksizeBits; + } + block.tokens.reserve(blocksize); + checkEOB = false; + } else { + block.tokens.reserve(1 << (6 + params.memLevel)); + } + while ((checkEOB && !predictEOB()) + || (!checkEOB && currentTokenCount < blocksize)) { + PreflateToken predictedToken = predictToken(); + if (predictedToken.len == 1) { + unsigned notok = codec->decode(CORR_LIT_MISPREDICTION); + if (!notok) { + block.tokens.push_back(predictedToken); + commitToken(predictedToken); + ++currentTokenCount; + continue; + } + if (!repredictReference(predictedToken)) { + predictionFailure = true; + return PreflateTokenBlock(); + } + } else { + unsigned notok = codec->decode(CORR_REF_MISPREDICTION); + if (notok) { + predictedToken.len = 1; + predictedToken.dist = 0; + block.tokens.push_back(predictedToken); + commitToken(predictedToken); + ++currentTokenCount; + continue; + } + } + bool lennotok = codec->decode(CORR_LEN_MISPREDICTION); + if (lennotok) { + int lendiff = codec->decode(CORR_LEN_CORRECTION, predictedToken.len); + int hops = codec->decode(CORR_DIST_AFTER_LEN_CORRECTION); + predictedToken.len += lendiff; + predictedToken.dist = state.firstMatch(predictedToken.len); + if (hops) { + predictedToken.dist = recalculateDistance(predictedToken, hops); + } + if (predictedToken.len < 3 || predictedToken.len > 258 + || predictedToken.dist == 0) { + predictionFailure = true; + return PreflateTokenBlock(); + } + } else { + bool distnotok = codec->decode(CORR_DIST_ONLY_MISPREDICTION); + if (distnotok) { + int hops = codec->decode(CORR_DIST_ONLY_CORRECTION); + if (hops) { + predictedToken.dist = recalculateDistance(predictedToken, hops); + } + if (predictedToken.dist == 0) { + predictionFailure = true; + return PreflateTokenBlock(); + } + } + } + block.tokens.push_back(predictedToken); + commitToken(predictedToken); + ++currentTokenCount; + } + return block; +} +bool PreflateTokenPredictor::decodeEOF(PreflateStatisticalDecoder* codec) { + if (state.availableInputSize() == 0) { + return codec->decodeValue(1) == 0; + } + return false; +} diff --git a/contrib/preflate/preflate_token_predictor.h b/contrib/preflate/preflate_token_predictor.h new file mode 100644 index 0000000..b647fc6 --- /dev/null +++ b/contrib/preflate/preflate_token_predictor.h @@ -0,0 +1,72 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_TOKEN_PREDICTOR_H +#define PREFLATE_TOKEN_PREDICTOR_H + +#include + +#include "preflate_parameter_estimator.h" +#include "preflate_predictor_state.h" +#include "preflate_statistical_codec.h" + +struct PreflateStatisticalModel; +struct PreflateStatisticalCodec; + +struct PreflateTokenPredictor { + PreflatePredictorState state; + PreflateHashChainExt hash; + PreflateParameters params; + bool predictionFailure; + bool fast; + unsigned prevLen; + PreflateToken pendingToken; + unsigned currentTokenCount; + bool emptyBlockAtEnd; + + struct BlockAnalysisResult { + PreflateTokenBlock::Type type; + unsigned tokenCount; + bool blockSizePredicted; + bool inputEOF; + bool lastBlock; + std::vector tokenInfo; + std::vector correctives; + }; + std::vector analysisResults; + + PreflateTokenPredictor(const PreflateParameters& params, + const std::vector& dump); + void analyzeBlock(const unsigned blockno, + const PreflateTokenBlock& block); + void updateModel(PreflateStatisticalModel*, + const unsigned blockno); + void encodeBlock(PreflateStatisticalEncoder*, + const unsigned blockno); + void encodeEOF(PreflateStatisticalEncoder*, + const unsigned blockno, + const bool lastBlock); + + PreflateTokenBlock decodeBlock(PreflateStatisticalDecoder*); + bool decodeEOF(PreflateStatisticalDecoder*); + + bool predictEOB(); + PreflateToken predictToken(); + bool repredictReference(PreflateToken& token); + PreflateRematchInfo repredictMatch(const PreflateToken&); + unsigned recalculateDistance(const PreflateToken&, const unsigned hops); + void commitToken(const PreflateToken&); +}; + +#endif /* PREFLATE_TOKEN_PREDICTOR_H */ diff --git a/contrib/preflate/preflate_tree_predictor.cpp b/contrib/preflate/preflate_tree_predictor.cpp new file mode 100644 index 0000000..f7dee3f --- /dev/null +++ b/contrib/preflate/preflate_tree_predictor.cpp @@ -0,0 +1,675 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "preflate_constants.h" +#include "preflate_statistical_codec.h" +#include "preflate_statistical_model.h" +#include "preflate_tree_predictor.h" + +PreflateTreePredictor::PreflateTreePredictor( + const std::vector& dump) + : input(dump) + , curPos(0) + , predictionFailure(false) { +} + +struct FreqIdxPair { + unsigned freq; + unsigned idx; +}; +struct TreeNode { + unsigned parent; + unsigned idx; +}; + +/* =========================================================================== +* Compares to subtrees, using the tree depth as tie breaker when +* the subtrees have equal frequency. This minimizes the worst case length. +*/ +bool pq_smaller(const FreqIdxPair& p1, const FreqIdxPair& p2, const unsigned char* nodeDepth) { + return p1.freq < p2.freq || (p1.freq == p2.freq && nodeDepth[p1.idx] <= nodeDepth[p2.idx]); +} + +/* =========================================================================== +* Restore the heap property by moving down the tree starting at node k, +* exchanging a node with the smallest of its two sons if necessary, stopping +* when the heap property is re-established (each father smaller than its +* two sons). +*/ +void pq_downheap(FreqIdxPair* ptr, const unsigned index, const unsigned len, const unsigned char* depth) { + unsigned k = index; + FreqIdxPair v = ptr[k]; + unsigned j = k * 2 + 1; /* left son of k */ + while (j < len) { + /* Set j to the smallest of the two sons: */ + if (j + 1 < len && pq_smaller(ptr[j + 1], ptr[j], depth)) { + j++; + } + /* Exit if v is smaller than both sons */ + if (pq_smaller(v, ptr[j], depth)) break; + + /* Exchange v with the smallest son */ + ptr[k] = ptr[j]; + k = j; + + /* And continue down the tree, setting j to the left son of k */ + j = k * 2 + 1; + } + ptr[k] = v; +} + +void pq_makeheap(FreqIdxPair* ptr, const unsigned len, const unsigned char* depth) { + for (unsigned n = (len - 1) / 2 + 1; n > 0; n--) { + pq_downheap(ptr, n - 1, len, depth); + } +} + +FreqIdxPair pq_remove(FreqIdxPair* ptr, unsigned& len, const unsigned char* depth) { + FreqIdxPair result = ptr[0]; + ptr[0] = ptr[--len]; + pq_downheap(ptr, 0, len, depth); + return result; +} + +unsigned PreflateTreePredictor::calcBitLengths( + unsigned char* symBitLen, + const unsigned* symFreq, + const unsigned symCount, + const unsigned maxBits, + const unsigned minMaxCode) { + FreqIdxPair toSort[PreflateConstants::L_CODES]; + TreeNode nodes[PreflateConstants::L_CODES * 2 + 1]; + unsigned char nodeBitLen[PreflateConstants::L_CODES * 2 + 1]; + unsigned char nodeDepth[PreflateConstants::L_CODES * 2 + 1]; + memset(nodeBitLen, 0, sizeof(nodeBitLen)); + memset(nodeDepth, 0, sizeof(nodeDepth)); + unsigned maxCode = 0, len = 0, nodeCount = 0, nodeId = symCount; + for (unsigned i = 0; i < symCount; ++i) { + if (symFreq[i]) { + toSort[len++] = FreqIdxPair {symFreq[i], maxCode = i}; + } + } + if (len < 2) { + memset(symBitLen, 0, symCount); + symBitLen[maxCode] = 1; + symBitLen[maxCode < 2 ? ++maxCode : 0] = 1; + return std::max(minMaxCode, maxCode + 1); + } + + pq_makeheap(toSort, len, nodeDepth); + while (len > 1) { + FreqIdxPair least1 = pq_remove(toSort, len, nodeDepth); + FreqIdxPair least2 = toSort[0]; + toSort[0] = FreqIdxPair {least1.freq + least2.freq, nodeId}; + nodes[nodeCount++] = TreeNode {nodeId, least1.idx}; + nodes[nodeCount++] = TreeNode {nodeId, least2.idx}; + nodeDepth[nodeId] = std::max(nodeDepth[least1.idx], nodeDepth[least2.idx]) + 1; + // note? original code put new entry at top of heap, and moved it downwards + // while push_heap pushes it upwards + pq_downheap(toSort, 0, len, nodeDepth); + nodeId++; + } + unsigned overflow = 0; + unsigned bl_count[16]; + memset(bl_count, 0, sizeof(bl_count)); + unsigned orgNodeCount = nodeCount; + while (nodeCount-- > 0) { + unsigned char newLen = nodeBitLen[nodes[nodeCount].parent] + 1; + if (newLen > maxBits) { + newLen = maxBits; + ++overflow; + } + unsigned idx = nodes[nodeCount].idx; + nodeBitLen[idx] = newLen; + if (idx < symCount) { + bl_count[newLen]++; + } + } + + if (overflow) { + unsigned bits; + do { + for (bits = maxBits - 1; bl_count[bits] == 0; bits--) { + } + bl_count[bits]--; /* move one leaf down the tree */ + bl_count[bits + 1] += 2; /* move one overflow item as its brother */ + bl_count[maxBits]--; + /* The brother of the overflow item also moves one step up, + * but this does not affect bl_count[max_length] + */ + overflow -= 2; + } while (overflow > 0); + + for (bits = maxBits, nodeCount = orgNodeCount; nodeCount > 0; ) { + --nodeCount; + unsigned idx = nodes[nodeCount].idx; + if (idx >= symCount) { + continue; + } + while (bl_count[bits] == 0) { + bits--; + } + nodeBitLen[idx] = bits; + bl_count[bits]--; + } + } + memcpy(symBitLen, nodeBitLen, symCount); + return std::max(minMaxCode, maxCode + 1); +} + +TreeCodeType PreflateTreePredictor::predictCodeType(const unsigned char* symBitLen, + const unsigned symCount, + const bool first) { + unsigned char code = symBitLen[0]; + if (code == 0) { + unsigned char curlen = 1; + unsigned char maxCurLen = std::min(symCount, 11u); + while (curlen < maxCurLen && symBitLen[curlen] == 0) { + ++curlen; + } + if (curlen >= 11) { + return TCT_REPZL; + } + if (curlen >= 3) { + return TCT_REPZS; + } + return TCT_BITS; + } + if (!first && code == symBitLen[-1]) { + unsigned char curlen = 1; + unsigned char maxCurLen = std::min(symCount, 3u); + while (curlen < maxCurLen && symBitLen[curlen] == code) { + ++curlen; + } + if (curlen >= 3) { + return TCT_REP; + } + } + return TCT_BITS; +} +unsigned char PreflateTreePredictor::predictCodeData(const unsigned char* symBitLen, + const TreeCodeType type, + const unsigned symCount, + const bool first) { + unsigned char code = symBitLen[0]; + switch (type) { + default: + case TCT_BITS: + return code; + case TCT_REP: + { + unsigned char curlen = 3; + unsigned char maxCurLen = std::min(symCount, 6u); + while (curlen < maxCurLen && symBitLen[curlen] == code) { + ++curlen; + } + return curlen; + } + case TCT_REPZS: + case TCT_REPZL: + { + unsigned char curlen = type == TCT_REPZS ? 3 : 11; + unsigned char maxCurLen = std::min(symCount, type == TCT_REPZS ? 10u : 138u); + while (curlen < maxCurLen && symBitLen[curlen] == 0) { + ++curlen; + } + return curlen; + } + } +} + + +void PreflateTreePredictor::predictLDTrees( + BlockAnalysisResult& analysis, + unsigned* frequencies, + const unsigned char* symBitLen, + const unsigned symLCount, + const unsigned symDCount, + const unsigned char* targetCodes, + const unsigned targetCodeSize) { + memset(frequencies, 0, sizeof(unsigned) * PreflateConstants::BL_CODES); + const unsigned char* ptr = symBitLen; + const unsigned char* code = targetCodes; + unsigned codeSize = targetCodeSize; + unsigned count1 = symLCount; + unsigned count2 = symDCount; + bool first = true; + while (codeSize > 0) { + TreeCodeType targetTreeCodeType; + switch (code[0]) { + case 16: targetTreeCodeType = TCT_REP; break; + case 17: targetTreeCodeType = TCT_REPZS; break; + case 18: targetTreeCodeType = TCT_REPZL; break; + default: targetTreeCodeType = TCT_BITS; break; + } + if (codeSize < 2 && targetTreeCodeType != TCT_BITS) { + predictionFailure = true; + return; + } + TreeCodeType predictedTreeCodeType = predictCodeType(ptr, count1, first); + unsigned char info = predictedTreeCodeType | ((targetTreeCodeType != predictedTreeCodeType) << 2); + if (targetTreeCodeType != predictedTreeCodeType) { + analysis.correctives.push_back(targetTreeCodeType); + } + unsigned char targetTreeCodeData = code[targetTreeCodeType != TCT_BITS]; + unsigned l = 1 + (targetTreeCodeType != TCT_BITS); + code += l; + codeSize -= l; + unsigned char predictedTreeCodeData = predictCodeData(ptr, targetTreeCodeType, count1, first); + first = false; + if (targetTreeCodeType != TCT_BITS) { + if (targetTreeCodeData != predictedTreeCodeData) { + info |= 8; + analysis.correctives.push_back(targetTreeCodeData); + } + } else { + analysis.correctives.push_back(predictedTreeCodeData); + analysis.correctives.push_back(targetTreeCodeData - predictedTreeCodeData); + } + if (targetTreeCodeType != TCT_BITS) { + frequencies[targetTreeCodeType + 15]++; + l = targetTreeCodeData; + } else { + frequencies[targetTreeCodeData]++; + l = 1; + } + ptr += l; + if (count1 > l) { + count1 -= l; + } else { + count1 += count2; + count2 = 0; + first = true; + if (count1 >= l) { + count1 -= l; + } else { + predictionFailure = true; + return; + } + } + analysis.tokenInfo.push_back(info); + } + analysis.tokenInfo.push_back(0xff); + if (count1 + count2 != 0) { + predictionFailure = true; + } +} + +void PreflateTreePredictor::collectTokenStatistics( + unsigned Lcodes[], + unsigned Dcodes[], + unsigned& Lcount, + unsigned& Dcount, + const PreflateTokenBlock& block) { + memset(Lcodes, 0, sizeof(unsigned) * PreflateConstants::L_CODES); + memset(Dcodes, 0, sizeof(unsigned) * PreflateConstants::D_CODES); + Lcount = 0; + Dcount = 0; + for (unsigned i = 0, n = block.tokens.size(); i < n; ++i) { + PreflateToken targetToken = block.tokens[i]; + if (targetToken.len == 1) { + Lcodes[input.curChar()]++; + Lcount++; + input.advance(1); + } else { + Lcodes[PreflateConstants::LITERALS + 1 + PreflateConstants::LCode(targetToken.len)]++; + Lcount++; + Dcodes[PreflateConstants::DCode(targetToken.dist)]++; + Dcount++; + input.advance(targetToken.len); + } + } + Lcodes[256] = 1; +} +unsigned PreflateTreePredictor::buildLBitlenghs( + unsigned char bitLengths[], + unsigned Lcodes[]) { + return calcBitLengths(bitLengths, Lcodes, PreflateConstants::L_CODES, 15, PreflateConstants::LITERALS + 1); +} +unsigned PreflateTreePredictor::buildDBitlenghs( + unsigned char bitLengths[], + unsigned Dcodes[]) { + return calcBitLengths(bitLengths, Dcodes, PreflateConstants::D_CODES, 15, 0); +} +unsigned PreflateTreePredictor::buildTCBitlengths( + unsigned char (&simpleCodeTree)[PreflateConstants::BL_CODES], + unsigned (&BLfreqs)[PreflateConstants::BL_CODES]) { + memset(simpleCodeTree, 0, sizeof(simpleCodeTree)); + calcBitLengths(simpleCodeTree, BLfreqs, PreflateConstants::BL_CODES, 7, 0); + unsigned predictedCTreeSize = PreflateConstants::BL_CODES; + while (predictedCTreeSize > 4 + && simpleCodeTree[PreflateConstants::treeCodeOrderTable[predictedCTreeSize - 1]] == 0) { + --predictedCTreeSize; + } + return predictedCTreeSize; +} + +void PreflateTreePredictor::analyzeBlock( + const unsigned blockno, + const PreflateTokenBlock& block) { + if (blockno != analysisResults.size() || predictionFailure) { + return; + } + analysisResults.push_back(BlockAnalysisResult()); + BlockAnalysisResult& analysis = analysisResults[blockno]; + analysis.blockType = block.type; + if (analysis.blockType != PreflateTokenBlock::DYNAMIC_HUFF) { + return; + } + + unsigned Lcodes[PreflateConstants::L_CODES], Dcodes[PreflateConstants::D_CODES]; + unsigned Lcount = 0, Dcount = 0; + collectTokenStatistics(Lcodes, Dcodes, Lcount, Dcount, block); + + unsigned char bitLengths[PreflateConstants::LD_CODES]; + memset(bitLengths, 0, sizeof(bitLengths)); + unsigned predictedLTreeSize = buildLBitlenghs(bitLengths, Lcodes); + analysis.tokenInfo.push_back(predictedLTreeSize != block.nlen); + if (predictedLTreeSize != block.nlen) { + analysis.correctives.push_back(block.nlen); + } + predictedLTreeSize = block.nlen; + + unsigned predictedDTreeSize = buildDBitlenghs(bitLengths + predictedLTreeSize, Dcodes); + analysis.tokenInfo.push_back(predictedDTreeSize != block.ndist); + if (predictedDTreeSize != block.ndist) { + analysis.correctives.push_back(block.ndist); + } + predictedDTreeSize = block.ndist; + + unsigned BLfreqs[PreflateConstants::BL_CODES]; + const unsigned char* targetCodes = &block.treecodes[0]; + unsigned targetCodeSize = block.treecodes.size(); + predictLDTrees(analysis, BLfreqs, bitLengths, predictedLTreeSize, predictedDTreeSize, targetCodes + block.ncode, targetCodeSize - block.ncode); + + unsigned char simpleCodeTree[PreflateConstants::BL_CODES]; + unsigned predictedCTreeSize = buildTCBitlengths(simpleCodeTree, BLfreqs); + analysis.tokenInfo.push_back(block.ncode); + analysis.tokenInfo.push_back(predictedCTreeSize != block.ncode); + predictedCTreeSize = block.ncode; + for (unsigned i = 0; i < predictedCTreeSize; ++i) { + unsigned predictedBL = simpleCodeTree[PreflateConstants::treeCodeOrderTable[i]]; + analysis.correctives.push_back(predictedBL); + analysis.correctives.push_back(targetCodes[i] - predictedBL); + } +} +void PreflateTreePredictor::encodeBlock( + PreflateStatisticalEncoder* codec, + const unsigned blockno) { + BlockAnalysisResult& analysis = analysisResults[blockno]; + if (analysis.blockType != PreflateTokenBlock::DYNAMIC_HUFF) { + return; + } + + unsigned infoPos = 0, correctivePos = 0; + unsigned char info = analysis.tokenInfo[infoPos++]; + codec->encode(CORR_L_COUNT_MISPREDICTION, info); + if (info) { + codec->encodeValue(analysis.correctives[correctivePos++] - PreflateConstants::LITERALS - 1, 5); + } + info = analysis.tokenInfo[infoPos++]; + codec->encode(CORR_D_COUNT_MISPREDICTION, info); + if (info) { + codec->encodeValue(analysis.correctives[correctivePos++], 5); + } + + while ((info = analysis.tokenInfo[infoPos++]) != 0xff) { + unsigned type = (info & 3); + codec->encode(CORR_LD_TYPE_MISPREDICTION, (info & 4) != 0, type); + if (info & 4) { + unsigned newType = analysis.correctives[correctivePos++]; + codec->encode(CORR_LD_TYPE_REPLACEMENT, newType, type); + type = newType; + } + if (type != TCT_BITS) { + codec->encode(CORR_LD_REPEAT_MISPREDICTION, (info & 8) != 0); + if (info & 8) { + unsigned newRepeat = analysis.correctives[correctivePos++]; + switch (type) { + case TCT_REP: + codec->encodeValue(newRepeat - 3, 2); + break; + case TCT_REPZS: + codec->encodeValue(newRepeat - 3, 3); + break; + case TCT_REPZL: + codec->encodeValue(newRepeat - 11, 7); + break; + } + } + } else { + unsigned bl_pred = analysis.correctives[correctivePos++]; + int bl_diff = analysis.correctives[correctivePos++]; + codec->encode(CORR_LD_BITLENGTH_CORRECTION, bl_diff, bl_pred); + } + } + unsigned blcount = analysis.tokenInfo[infoPos++]; + info = analysis.tokenInfo[infoPos++]; + codec->encode(CORR_TC_COUNT_MISPREDICTION, info); + if (info) { + codec->encodeValue(blcount - 4, 4); + } + for (unsigned i = 0; i < blcount; ++i) { + int bl_pred = analysis.correctives[correctivePos++]; + int bl_diff = analysis.correctives[correctivePos++]; + codec->encode(CORR_TC_BITLENGTH_CORRECTION, bl_diff, bl_pred); + } +} + +void PreflateTreePredictor::updateModel( + PreflateStatisticalModel* model, + const unsigned blockno) { + BlockAnalysisResult& analysis = analysisResults[blockno]; + if (analysis.blockType != PreflateTokenBlock::DYNAMIC_HUFF) { + return; + } + + unsigned infoPos = 0, correctivePos = 0; + unsigned char info = analysis.tokenInfo[infoPos++]; + model->LCountMisprediction[info]++; + if (info) { + correctivePos++; + } + info = analysis.tokenInfo[infoPos++]; + model->DCountMisprediction[info]++; + if (info) { + correctivePos++; + } + + while ((info = analysis.tokenInfo[infoPos++]) != 0xff) { + unsigned type = (info & 3); + model->LDTypeMisprediction[type][(info & 4) != 0]++; + if (info & 4) { + unsigned newType = analysis.correctives[correctivePos++]; + model->LDTypeReplacement[newType]++; + type = newType; + } + if (type != TCT_BITS) { + model->LDRepeatCountMisprediction[(info & 8) != 0]++; + if (info & 8) { + correctivePos++; + } + } else { + /*unsigned bl_pred = analysis.correctives[*/correctivePos++/*]*/; + int bl_diff = analysis.correctives[correctivePos++]; + if (bl_diff >= 0) { + model->LDBitlengthPositiveCorrection[std::min(4, bl_diff)]++; + } else { + model->LDBitlengthNegativeCorrection[std::min(3, -bl_diff - 1)]++; + } + } + } + unsigned blcount = analysis.tokenInfo[infoPos++]; + info = analysis.tokenInfo[infoPos++]; + model->TCCountMisprediction[info]++; + for (unsigned i = 0; i < blcount; ++i) { + /*int bl_pred = analysis.correctives[*/correctivePos++/*]*/; + int bl_diff = analysis.correctives[correctivePos++]; + if (bl_diff >= 0) { + model->TCBitlengthPositiveCorrection[std::min(3, bl_diff)]++; + } else { + model->TCBitlengthNegativeCorrection[std::min(2, -bl_diff - 1)]++; + } + } +} + + +unsigned PreflateTreePredictor::reconstructLDTrees( + PreflateStatisticalDecoder* codec, + unsigned* frequencies, + unsigned char* targetCodes, + const unsigned targetCodeSize, + const unsigned char* symBitLen, + const unsigned symLCount, + const unsigned symDCount) { + memset(frequencies, 0, sizeof(unsigned) * PreflateConstants::BL_CODES); + const unsigned char* ptr = symBitLen; + unsigned osize = 0; + unsigned count1 = symLCount; + unsigned count2 = symDCount; + bool first = true; + while (count1 + count2 > 0) { + TreeCodeType predictedTreeCodeType = predictCodeType(ptr, count1, first); + if (codec->decode(CORR_LD_TYPE_MISPREDICTION, predictedTreeCodeType)) { + unsigned newType = codec->decode(CORR_LD_TYPE_REPLACEMENT, predictedTreeCodeType); + switch (newType) { + case TCT_BITS: + predictedTreeCodeType = TCT_BITS; + break; + case TCT_REP: + predictedTreeCodeType = TCT_REP; + break; + case TCT_REPZS: + predictedTreeCodeType = TCT_REPZS; + break; + case TCT_REPZL: + predictedTreeCodeType = TCT_REPZL; + break; + } + } + unsigned char predictedTreeCodeData = predictCodeData(ptr, predictedTreeCodeType, count1, first); + first = false; + if (predictedTreeCodeType != TCT_BITS) { + if (codec->decode(CORR_LD_REPEAT_MISPREDICTION)) { + switch (predictedTreeCodeType) { + case TCT_REP: + predictedTreeCodeData = codec->decodeValue(2) + 3; + break; + case TCT_REPZS: + predictedTreeCodeData = codec->decodeValue(3) + 3; + break; + case TCT_REPZL: + predictedTreeCodeData = codec->decodeValue(7) + 11; + break; + case TCT_BITS: + // unreachable + break; + } + } + } else { + int bl_diff = codec->decode(CORR_LD_BITLENGTH_CORRECTION, predictedTreeCodeData); + predictedTreeCodeData += bl_diff; + } + unsigned l; + if (predictedTreeCodeType != TCT_BITS) { + frequencies[predictedTreeCodeType + 15]++; + l = predictedTreeCodeData; + if (osize + 2 > targetCodeSize) { + predictionFailure = true; + break; + } + targetCodes[osize++] = predictedTreeCodeType + 15; + targetCodes[osize++] = predictedTreeCodeData; + } else { + frequencies[predictedTreeCodeData]++; + l = 1; + if (osize >= targetCodeSize) { + predictionFailure = true; + break; + } + targetCodes[osize++] = predictedTreeCodeData; + } + ptr += l; + if (count1 > l) { + count1 -= l; + } else { + count1 += count2; + count2 = 0; + first = true; + if (count1 >= l) { + count1 -= l; + } else { + predictionFailure = true; + break; + } + } + } + if (count1 + count2 != 0) { + predictionFailure = true; + } + return predictionFailure ? 0 : osize; +} + +bool PreflateTreePredictor::decodeBlock( + PreflateTokenBlock& block, + PreflateStatisticalDecoder* codec) { + if (block.type != PreflateTokenBlock::DYNAMIC_HUFF) { + return true; + } + + unsigned Lcodes[PreflateConstants::L_CODES], Dcodes[PreflateConstants::D_CODES]; + unsigned Lcount = 0, Dcount = 0; + collectTokenStatistics(Lcodes, Dcodes, Lcount, Dcount, block); + + unsigned char bitLengths[PreflateConstants::LD_CODES]; + memset(bitLengths, 0, sizeof(bitLengths)); + unsigned predictedLTreeSize = buildLBitlenghs(bitLengths, Lcodes); + if (codec->decode(CORR_L_COUNT_MISPREDICTION)) { + predictedLTreeSize = codec->decodeValue(5) + PreflateConstants::LITERALS + 1; + } + block.nlen = predictedLTreeSize; + + unsigned predictedDTreeSize = buildDBitlenghs(bitLengths + predictedLTreeSize, Dcodes); + if (codec->decode(CORR_D_COUNT_MISPREDICTION)) { + predictedDTreeSize = codec->decodeValue(5); + } + block.ndist = predictedDTreeSize; + + unsigned BLfreqs[PreflateConstants::BL_CODES]; + unsigned char compressedLDtrees[PreflateConstants::LD_CODES]; + unsigned targetCodeSize = reconstructLDTrees(codec, BLfreqs, compressedLDtrees, PreflateConstants::LD_CODES, + bitLengths, predictedLTreeSize, predictedDTreeSize); + if (predictionFailure) { + return false; + } + + unsigned char simpleCodeTree[PreflateConstants::BL_CODES]; + unsigned predictedCTreeSize = buildTCBitlengths(simpleCodeTree, BLfreqs); + if (codec->decode(CORR_TC_COUNT_MISPREDICTION)) { + predictedCTreeSize = codec->decodeValue(4) + 4; + } + block.ncode = predictedCTreeSize; + unsigned char shuffledCodeTree[PreflateConstants::BL_CODES]; + for (unsigned i = 0; i < predictedCTreeSize; ++i) { + unsigned predictedBL = simpleCodeTree[PreflateConstants::treeCodeOrderTable[i]]; + int bl_diff = codec->decode(CORR_TC_BITLENGTH_CORRECTION, predictedBL); + shuffledCodeTree[i] = predictedBL + bl_diff; + } + block.treecodes.reserve(predictedCTreeSize + targetCodeSize); + block.treecodes.insert(block.treecodes.end(), shuffledCodeTree, shuffledCodeTree + predictedCTreeSize); + block.treecodes.insert(block.treecodes.end(), compressedLDtrees, compressedLDtrees + targetCodeSize); + return true; +} diff --git a/contrib/preflate/preflate_tree_predictor.h b/contrib/preflate/preflate_tree_predictor.h new file mode 100644 index 0000000..0343801 --- /dev/null +++ b/contrib/preflate/preflate_tree_predictor.h @@ -0,0 +1,99 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_TREE_PREDICTOR_H +#define PREFLATE_TREE_PREDICTOR_H + +#include + +#include "preflate_constants.h" +#include "preflate_input.h" +#include "preflate_parameter_estimator.h" + +struct PreflateStatisticalModel; +struct PreflateStatisticalCodec; + +enum TreeCodeType { + TCT_BITS = 0, TCT_REP = 1, TCT_REPZS = 2, TCT_REPZL = 3 +}; + +struct PreflateTreePredictor { + PreflateInput input; + unsigned curPos; + bool predictionFailure; + + struct BlockAnalysisResult { + PreflateTokenBlock::Type blockType; + std::vector tokenInfo; + std::vector correctives; + }; + std::vector analysisResults; + + void collectTokenStatistics( + unsigned LcodeFrequencies[], + unsigned DcodeFrequencies[], + unsigned& Lcount, + unsigned& Dcount, + const PreflateTokenBlock& block); + unsigned buildLBitlenghs( + unsigned char bitLengths[], + unsigned Lcodes[]); + unsigned buildDBitlenghs( + unsigned char bitLengths[], + unsigned Dcodes[]); + unsigned buildTCBitlengths( + unsigned char (&bitLengths)[PreflateConstants::BL_CODES], + unsigned (&BLfreqs)[PreflateConstants::BL_CODES]); + + + unsigned calcBitLengths(unsigned char* symBitLen, + const unsigned* symFreq, + const unsigned symCount, + const unsigned maxBits, + const unsigned minMaxCode); + + TreeCodeType predictCodeType(const unsigned char* symBitLen, + const unsigned symCount, + const bool first); + unsigned char predictCodeData(const unsigned char* symBitLen, + const TreeCodeType type, + const unsigned symCount, + const bool first); + void predictLDTrees(BlockAnalysisResult& analysis, + unsigned* frequencies, + const unsigned char* symBitLen, + const unsigned symLCount, + const unsigned symDCount, + const unsigned char* targetCodes, + const unsigned targetCodeSize); + unsigned reconstructLDTrees(PreflateStatisticalDecoder* codec, + unsigned* frequencies, + unsigned char* targetCodes, + unsigned targetCodeSize, + const unsigned char* symBitLen, + const unsigned symLCount, + const unsigned symDCount); + + PreflateTreePredictor(const std::vector& dump); + void analyzeBlock(const unsigned blockno, + const PreflateTokenBlock& block); + void updateModel(PreflateStatisticalModel*, + const unsigned blockno); + void encodeBlock(PreflateStatisticalEncoder*, + const unsigned blockno); + + bool decodeBlock(PreflateTokenBlock& block, PreflateStatisticalDecoder*); +}; + +#endif /* PREFLATE_TREE_PREDICTOR_H */ diff --git a/contrib/preflate/preflate_unpack.cpp b/contrib/preflate/preflate_unpack.cpp new file mode 100644 index 0000000..af889a9 --- /dev/null +++ b/contrib/preflate/preflate_unpack.cpp @@ -0,0 +1,69 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "preflate_dumper.h" +#include "preflate_unpack.h" +#include "zlib1.2.11.dec/zlib.h" + +void* zalloc(void*, uInt items, uInt size) { + return malloc(items * size); +} +void zfree(void*, void* addr) { + free(addr); +} + +bool preflate_unpack(std::vector& unpacked_output, + std::vector& blocks, + const std::vector& deflate_raw) { + z_stream inf; /* zlib deflate and inflate states */ + inf.zalloc = zalloc; + inf.zfree = zfree; + inf.opaque = Z_NULL; + inf.avail_in = deflate_raw.size(); + inf.next_in = const_cast(deflate_raw.data()); + + PreflateDumper dumper; + inf.dumper = &dumper; + int ret = inflateInit2(&inf, -15); + if (ret != Z_OK) { + return false; + } + std::vector tmp(64 * 1024); + inf.avail_out = tmp.size(); + inf.next_out = tmp.data(); + bool failure = false; + + do { + /* decompress */ + inf.avail_out = tmp.size(); + inf.next_out = tmp.data(); + ret = inflate(&inf, Z_NO_FLUSH); + if (ret == Z_STREAM_ERROR || ret == Z_DATA_ERROR || ret == Z_NEED_DICT || ret == Z_MEM_ERROR + || ret == Z_BUF_ERROR) { + return false; + } + dumper.uncompressed.insert(dumper.uncompressed.end(), tmp.begin(), tmp.end() - inf.avail_out); + } while (ret != Z_STREAM_END); + + inflateEnd(&inf); + + if (failure || dumper.hadErrors()) { + return false; + } + + unpacked_output = std::move(dumper.uncompressed); + blocks = std::move(dumper.blocks); + return true; +} diff --git a/contrib/preflate/preflate_unpack.h b/contrib/preflate/preflate_unpack.h new file mode 100644 index 0000000..76d19f2 --- /dev/null +++ b/contrib/preflate/preflate_unpack.h @@ -0,0 +1,25 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef PREFLATE_UNPACK_H +#define PREFLATE_UNPACK_H + +#include +#include "preflate_token.h" + +bool preflate_unpack(std::vector& unpacked_output, + std::vector& blocks, + const std::vector& deflate_raw); + +#endif /* PREFLATE_UNPACK_H */ diff --git a/contrib/preflate/support/arithmetic_coder.cpp b/contrib/preflate/support/arithmetic_coder.cpp new file mode 100644 index 0000000..5e54afc --- /dev/null +++ b/contrib/preflate/support/arithmetic_coder.cpp @@ -0,0 +1,212 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "arithmetic_coder.h" +#include "array_helper.h" +#include "bit_helper.h" + +ArithmeticEncoder::ArithmeticEncoder(BitOutputStream& bos) + : _bos(bos) + , _low(0) + , _high(0x7fffffff) + , _e3cnt(0) {} + +void ArithmeticEncoder::_writeE3(const unsigned w) { + while (_e3cnt > 0) { + uint32_t todo = std::min(_e3cnt, 16u); + _bos.put(w, todo); + _e3cnt -= todo; + } +} + +void ArithmeticEncoder::flush() { + if (_low < 0x20000000) { // case a.) + _bos.put(2, 2); // write 0, 1, E3 + _writeE3(~0u); + } else { + _bos.put(1, 1); + } +} + +void ArithmeticEncoder::_normalize() { + // write determinated bits + // this is the case if _low features 1 bits + // or _high features 0 bits + uint32_t lh = ~_low & _high; + if ((lh & 0x40000000) == 0) { + unsigned w = (_low & 0x40000000) != 0; + _bos.put(w, 1); + _writeE3(w - 1); + if ((lh & 0x20000000) == 0) { + unsigned l = bitLeadingZeroes((lh << 2) + 3); + if (l <= 16) { + _bos.putReverse(_low >> (30 - l), l); + } else { + _bos.putReverse(_low >> (30 - 16), 16); + _bos.putReverse(_low >> (30 - l), l - 16); + } + _low = (_low << (l + 1)) & 0x7fffffff; + _high = (((_high + 1) << (l + 1)) - 1) & 0x7fffffff; + } else { + _low = (_low << 1) & 0x7fffffff; + _high = ((_high << 1) + 1) & 0x7fffffff; + } + } + + // count indeterminated bits + lh = ~_low | _high; + if ((lh & 0x20000000) == 0) { + // low starts with 01, high starts with 10 + unsigned l = bitLeadingZeroes((lh << 2) + 3); + _e3cnt += l; + _low = (_low << l) & 0x3fffffff; + _high = ((((_high + 1) << l) - 1) & 0x3fffffff) + | 0x40000000; + } +} + +ArithmeticDecoder::ArithmeticDecoder(BitInputStream& bis) + : _bis(bis) + , _value(0) + , _low(0) + , _high(0x7fffffff) { + _value = _bis.getReverse(16) << 15; + _value |= _bis.getReverse(15); +} +void ArithmeticDecoder::_normalize() { + // skip determinated bits + // this is the case if _low features 1 bits + // or _high features 0 bits + uint32_t lh = ~_low & _high; + if ((lh & 0x40000000) == 0) { + //unsigned w = (_low & 0x40000000) != 0; + if ((lh & 0x20000000) == 0) { + unsigned l = bitLeadingZeroes((lh << 2) + 3); + _low = (_low << (l + 1)) & 0x7fffffff; + _high = (((_high + 1) << (l + 1)) - 1) & 0x7fffffff; + if (l <= 15) { + _value = ((_value << (l + 1)) + _bis.getReverse(l + 1)) & 0x7fffffff; + } else { + _value = ((_value << 16) + _bis.getReverse(16)) & 0x7fffffff; + _value = ((_value << (l - 15)) + _bis.getReverse(l - 15)) & 0x7fffffff; + } + } else { + _low = (_low << 1) & 0x7fffffff; + _high = ((_high << 1) + 1) & 0x7fffffff; + _value = ((_value << 1) + _bis.get(1)) & 0x7fffffff; + } + } + + // count indeterminated bits + lh = ~_low | _high; + if ((lh & 0x20000000) == 0) { + // low starts with 01, high starts with 10 + unsigned l = bitLeadingZeroes((lh << 2) + 3); + _low = (_low << l) & 0x3fffffff; + _high = ((((_high + 1) << l) - 1) & 0x3fffffff) + | 0x40000000; + if (l <= 16) { + _value = ((_value << l) + _bis.getReverse(l)) & 0x7fffffff; + } else { + _value = ((_value << 16) + _bis.getReverse(16)) & 0x7fffffff; + _value = ((_value << (l - 16)) + _bis.getReverse(l - 16)) & 0x7fffffff; + } + } +} + +bool modelCheckFixed(unsigned bounds[], unsigned short ids[], unsigned short rids[], + const unsigned N) { + unsigned idx = N; + for (unsigned i = 0; i < N; ++i) { + if (bounds[i]) { + if (idx != N) { + return false; + } + idx = i; + } + } + ids[N - 1] = idx; + rids[idx] = N - 1; + bounds[idx] = 0; + bounds[N] = 1 << 16; + return true; +} + +void modelSortBounds(unsigned bounds[], unsigned short ids[], unsigned short rids[], + unsigned backup[], const unsigned N) { + for (unsigned i = 0; i < N; ++i) { + ids[i] = i; + backup[i] = bounds[i]; + } + std::sort(ids, ids + N, [=](unsigned i1, unsigned i2) { + if (backup[i1] != backup[i2]) { + return backup[i1] < backup[i2]; + } + return i1 < i2; + }); + for (unsigned i = 0; i < N; ++i) { + bounds[i] = backup[ids[i]]; + rids[ids[i]] = i; + } +} + +void modelRecreateBounds(unsigned bounds[], const unsigned N) { + unsigned sum = sumArray(bounds, N), acc, prev; + prev = bounds[0]; + bounds[0] = acc = 0; + for (unsigned i = 0; i < N; ++i) { + if (prev) { + acc += prev; + prev = bounds[i + 1]; + int diff = (((uint64_t)acc) << 16) / sum - bounds[i]; + unsigned diff_bits = bitLength(diff); + const unsigned k = 5; + if (diff > 0 && diff_bits > k) { + diff = diff & (((1 << k) - 1) << (diff_bits - k)); + } + bounds[i + 1] = bounds[i] + diff; + if (bounds[i + 1] <= bounds[i]) { + bounds[i + 1] = bounds[i] + 1; + } + } else { + prev = bounds[i + 1]; + bounds[i + 1] = bounds[i]; + } + } + if (bounds[N] > 0) { + bounds[N] = 1 << 16; + } +} + +void ACFixedScaleBinaryModel::build() { + if (bounds[0] == 0 || bounds[1] == 0) { + _fixed = true; + ids[1] = bounds[0] == 0; + rids[ids[1]] = 1; + bounds[1] = bounds[0] = 0; + bounds[2] = 1 << 16; + return; + } + ids[0] = 0; + ids[1] = 1; + if (bounds[1] < bounds[0]) { + std::swap(ids[0], ids[1]); + std::swap(bounds[0], bounds[1]); + } + rids[ids[0]] = 0; + rids[ids[1]] = 1; + + modelRecreateBounds(bounds, 2); +} diff --git a/contrib/preflate/support/arithmetic_coder.h b/contrib/preflate/support/arithmetic_coder.h new file mode 100644 index 0000000..6e5d818 --- /dev/null +++ b/contrib/preflate/support/arithmetic_coder.h @@ -0,0 +1,224 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef ARITHMETIC_CODER_H +#define ARITHMETIC_CODER_H + +#include +#include +#include "bitstream.h" +#include "const_division.h" + +class ArithmeticEncoder { +public: + ArithmeticEncoder(BitOutputStream& bos); + void flush(); + void encode(const uint32_t scale, const uint32_t low, const uint32_t high) { + // update steps, low count, high count + uint32_t step = ((_high - _low) + 1) / scale; + _high = _low + step * high - 1; + _low += step * low; + _normalize(); + } + void encodeShiftScale(const uint32_t shift, const uint32_t low, const uint32_t high) { + // update steps, low count, high count + uint32_t step = ((_high - _low) + 1) >> shift; + _high = _low + step * high - 1; + _low += step * low; + _normalize(); + } + void encode(const udivider_t<32>& scale, const uint32_t low, const uint32_t high) { + // update steps, low count, high count + uint32_t step = divide((_high - _low) + 1, scale); + _high = _low + step * high - 1; + _low += step * low; + _normalize(); + } + +private: + void _normalize(); + void _writeE3(const unsigned w); + + BitOutputStream& _bos; + + // arithmetic coding variables + uint32_t _low; + uint32_t _high; + uint32_t _e3cnt; +}; + +class ArithmeticDecoder { +public: + ArithmeticDecoder(BitInputStream& bis); + unsigned decode(const uint32_t scale, const unsigned bounds[], const unsigned N) { + uint32_t step = ((_high - _low) + 1) / scale; + return _decode(step, bounds, N); + } + unsigned decodeShiftScale(const uint32_t shift, const unsigned bounds[], const unsigned N) { + uint32_t step = ((_high - _low) + 1) >> shift; + return _decode(step, bounds, N); + } + unsigned decode(const udivider_t<32>& scale, const unsigned bounds[], const unsigned N) { + uint32_t step = divide((_high - _low) + 1, scale); + return _decode(step, bounds, N); + } + + unsigned decodeBinary(const uint32_t scale, const unsigned bounds[]) { + uint32_t step = ((_high - _low) + 1) / scale; + return _decodeBinary(step, bounds); + } + unsigned decodeBinaryShiftScale(const uint32_t shift, const unsigned bounds[]) { + uint32_t step = ((_high - _low) + 1) >> shift; + return _decodeBinary(step, bounds); + } + unsigned decodeBinary(const udivider_t<32>& scale, const unsigned bounds[]) { + uint32_t step = divide((_high - _low) + 1, scale); + return _decodeBinary(step, bounds); + } + +private: + unsigned _findIndex(const unsigned bounds[], + const unsigned N, + const unsigned val) { + for (unsigned i = 0; i < N; ++i) { + if (val < bounds[i + 1]) { + return i; + } + } + return N - 1; + } + unsigned _decode(const uint32_t step, const unsigned bounds[], const unsigned N) { + uint32_t val = (_value - _low) / step; + unsigned result = _findIndex(bounds, N, val); + _high = _low + step * bounds[result + 1] - 1; + _low += step * bounds[result]; + _normalize(); + return result; + } + unsigned _decodeBinary(const uint32_t step, const unsigned bounds[]) { + unsigned result = (_value >= _low + bounds[1] * step); + _high = _low + step * bounds[result + 1] - 1; + _low += step * bounds[result]; + _normalize(); + return result; + } + void _normalize(); + + BitInputStream& _bis; + + // arithmetic coding variables + uint32_t _value; + uint32_t _low; + uint32_t _high; +}; + +bool modelCheckFixed(unsigned bounds[], unsigned short ids[], unsigned short rids[], + const unsigned N); +void modelSortBounds(unsigned bounds[], unsigned short ids[], unsigned short rids[], + unsigned backup[], const unsigned N); +void modelRecreateBounds(unsigned bounds[], const unsigned N); + +template +struct ACModelBase { + static const unsigned L = N; + bool isEqualTo(const ACModelBase& m) const { + for (unsigned i = 0; i < N; ++i) { + if (bounds[i] != m.bounds[i]) { + return false; + } + if (bounds[i + 1] > 0 && ids[i] != m.ids[i]) { + return false; + } + } + if (bounds[N] != m.bounds[N]) { + return false; + } + return true; + } + + unsigned bounds[N + 1]; + unsigned short ids[N], rids[N]; + bool _fixed; +}; + +struct ACFixedScaleBinaryModel : public ACModelBase<2> { + ACFixedScaleBinaryModel() {} + ACFixedScaleBinaryModel(const unsigned(&arr)[2]) { + memcpy(this->bounds, arr, sizeof(arr)); + build(); + } + void build(); + void encode(ArithmeticEncoder* encoder, const unsigned item) { + if (!this->_fixed) { + unsigned pos = this->rids[item]; + encoder->encodeShiftScale(16, this->bounds[pos], this->bounds[pos + 1]); + } + } +#if 0 + unsigned decode(aricoder* codec) { + symbol s; + s.scale = 1 << 16; + unsigned cnt = codec->decode_count(&s); + for (unsigned i = 0; i < N; ++i) { + if (cnt < bounds[i + 1]) { + s.low_count = bounds[i]; + s.high_count = bounds[i + 1]; + codec->decode(&s); + return ids[i]; + } + } + return 0; + } +#endif +}; + +template +struct ACFixedScaleModel : public ACModelBase { + ACFixedScaleModel() {} + ACFixedScaleModel(const unsigned(&arr)[N]) { + memcpy(this->bounds, arr, sizeof(arr)); + build(); + } + void build() { + unsigned backup[N]; + if (!(this->_fixed = modelCheckFixed(this->bounds, this->ids, this->rids, N))) { + modelSortBounds(this->bounds, this->ids, this->rids, backup, N); + modelRecreateBounds(this->bounds, N); + } + } + void encode(ArithmeticEncoder* encoder, const unsigned item) { + if (!this->_fixed) { + unsigned pos =this->rids[item]; + encoder->encodeShiftScale(16, this->bounds[pos], this->bounds[pos + 1]); + } + } +#if 0 + unsigned decode(aricoder* codec) { + symbol s; + s.scale = 1 << 16; + unsigned cnt = codec->decode_count(&s); + for (unsigned i = 0; i < N; ++i) { + if (cnt < bounds[i + 1]) { + s.low_count = bounds[i]; + s.high_count = bounds[i + 1]; + codec->decode(&s); + return this->ids[i]; + } + } + return 0; + } +#endif +}; + +#endif /* ARITHMETIC_CODER_H */ diff --git a/contrib/preflate/support/array_helper.cpp b/contrib/preflate/support/array_helper.cpp new file mode 100644 index 0000000..ae38d81 --- /dev/null +++ b/contrib/preflate/support/array_helper.cpp @@ -0,0 +1,23 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "array_helper.h" + +unsigned sumArray(const unsigned* data, const unsigned n) { + unsigned sum = 0; + for (unsigned i = 0; i < n; ++i) { + sum += data[i]; + } + return sum; +} diff --git a/contrib/preflate/support/array_helper.h b/contrib/preflate/support/array_helper.h new file mode 100644 index 0000000..e2400ab --- /dev/null +++ b/contrib/preflate/support/array_helper.h @@ -0,0 +1,25 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef ARRAY_HELPER_H +#define ARRAY_HELPER_H + +unsigned sumArray(const unsigned* data, const unsigned n); + +template +inline unsigned sumArray(const unsigned (&data)[N]) { + return sumArray(data, N); +} + +#endif /* ARRAY_HELPER_H */ diff --git a/contrib/preflate/support/bit_helper.cpp b/contrib/preflate/support/bit_helper.cpp new file mode 100644 index 0000000..fb1606f --- /dev/null +++ b/contrib/preflate/support/bit_helper.cpp @@ -0,0 +1,59 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "bit_helper.h" + +unsigned bitLength(unsigned value) { + unsigned l = 0; + while (value > 0) { + l++; + value >>= 1; + } + return l; +} + +static unsigned char reverse4[16] = {0, 8, 4, 12, 2, 10, 6, 14, 1, 9, 5, 13, 3, 11, 7, 15}; +static unsigned bitReverse8(const unsigned value) { + return (reverse4[value & 0x0f] << 4) | reverse4[(value >> 4) & 0x0f]; +} +static unsigned bitReverse16(const unsigned value) { + return (bitReverse8(value & 0xff) << 8) | bitReverse8(value >> 8); +} +static unsigned bitReverse32(const unsigned value) { + return (bitReverse16(value & 0xffff) << 16) | bitReverse16(value >> 16); +} +unsigned bitReverse(const unsigned value, const unsigned bits) { + if (bits <= 8) { + return bitReverse8(value) >> (8 - bits); + } + if (bits <= 16) { + return bitReverse16(value) >> (16 - bits); + } + return bitReverse32(value) >> (32 - bits); +} + +static unsigned char leading4[16] = {4, 3, 2, 2, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0}; +unsigned bitLeadingZeroes(const unsigned value_) { + if (value_ == 0) { + return 32; + } + unsigned value = value_; + unsigned result = 0; + while ((value & 0xf0000000) == 0) { + value <<= 4; + result += 4; + } + return result + leading4[value >> 28]; +} + diff --git a/contrib/preflate/support/bit_helper.h b/contrib/preflate/support/bit_helper.h new file mode 100644 index 0000000..ccee6db --- /dev/null +++ b/contrib/preflate/support/bit_helper.h @@ -0,0 +1,22 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef BIT_HELPER_H +#define BIT_HELPER_H + +unsigned bitLength(unsigned value); +unsigned bitReverse(const unsigned value, const unsigned bits); +unsigned bitLeadingZeroes(const unsigned value); + +#endif /* BIT_HELPER_H */ diff --git a/contrib/preflate/support/bitstream.cpp b/contrib/preflate/support/bitstream.cpp new file mode 100644 index 0000000..a8ff360 --- /dev/null +++ b/contrib/preflate/support/bitstream.cpp @@ -0,0 +1,120 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "bitstream.h" + +BitInputStream::BitInputStream(InputStream& is) + : _input(is) + , _bufPos(0) + , _bufSize(0) + , _bufFastLimit(0) + , _eof(false) + , _bits(0) + , _bitsRemaining(0) + , _totalBitPos(0) +{} + +void BitInputStream::_fillBytes() { + // free space in bit buffer + if (_bufPos >= _bufFastLimit) { + if (!_eof) { + unsigned remaining = _bufSize - _bufPos; + memcpy(_buffer + PRE_BUF_EXTRA - remaining, + _buffer + _bufPos, remaining); + _bufPos = PRE_BUF_EXTRA - remaining; + _bufSize = PRE_BUF_EXTRA + _input.read(_buffer + PRE_BUF_EXTRA, BUF_SIZE); + _bufFastLimit = std::max(_bufPos, _bufSize - PRE_BUF_EXTRA); + _eof = _bufSize != PRE_BUF_EXTRA + BUF_SIZE; + } + } +} +void BitInputStream::_fill() { + // free space in bit buffer + if (_bufPos >= _bufFastLimit) { + if (!_eof) { + _fillBytes(); + } + while (_bitsRemaining <= BITS - 8 && _bufPos < _bufSize) { + _bits |= ((size_t)_buffer[_bufPos++]) << _bitsRemaining; + _bitsRemaining += 8; + } + return; + } + while (_bitsRemaining <= BITS - 8) { + _bits |= ((size_t)_buffer[_bufPos++]) << _bitsRemaining; + _bitsRemaining += 8; + } +} +size_t BitInputStream::copyBytesTo(OutputStream& output, const size_t len) { + if (_bitsRemaining & 7) { + return 0; + } + uint8_t a[sizeof(_bits)]; + size_t l = 0; + while (_bitsRemaining > 0 && l < len) { + a[l++] = _bits & 0xff; + _bitsRemaining -= 8; + _bits >>= 8; + _totalBitPos += 8; + } + size_t w = output.write(a, l); + if (w != l) { + return w; + } + while (l < len) { + unsigned todo = std::min(len - l, (size_t)(_bufSize - _bufPos)); + w = output.write(_buffer + _bufPos, todo); + _totalBitPos += 8 * w; + _bufPos += w; + l += w; + if (w != todo) { + return l; + } + _fillBytes(); + } + return l; +} + +BitOutputStream::BitOutputStream(OutputStream& output) + : _output(output) + , _bufPos(0) + , _bits(0) + , _bitPos(0) {} + +void BitOutputStream::_flush() { + while (_bitPos >= 8) { + _buffer[_bufPos++] = _bits & 0xff; + _bits >>= 8; + _bitPos -= 8; + } + if (_bufPos >= BUF_SIZE) { + _output.write(_buffer, BUF_SIZE); + memcpy(_buffer, _buffer + BUF_SIZE, _bufPos - BUF_SIZE); + _bufPos -= BUF_SIZE; + } +} +void BitOutputStream::flush() { + _flush(); + + if (_bitPos > 0) { + _buffer[_bufPos++] = _bits & 0xff; + _bits = 0; + _bitPos = 0; + } + + _output.write(_buffer, _bufPos); + _bufPos = 0; +} diff --git a/contrib/preflate/support/bitstream.h b/contrib/preflate/support/bitstream.h new file mode 100644 index 0000000..e393cfd --- /dev/null +++ b/contrib/preflate/support/bitstream.h @@ -0,0 +1,126 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef BITSTREAM_H +#define BITSTREAM_H + +#include +#include "bit_helper.h" +#include "stream.h" + +// Huffman decoder for little endian +class BitInputStream { +public: + BitInputStream(InputStream&); + + bool eof() const { + return _eof && _bufPos == _bufSize && !_bitsRemaining; + } + + size_t bitPos() const { + return _totalBitPos; + } + + size_t peek(const unsigned n) { + if (_bitsRemaining < n) { + _fill(); + } + return _bits & ((1 << n) - 1); + } + void skip(const unsigned n) { + _bitsRemaining -= std::min(n, _bitsRemaining); + _bits >>= n; + _totalBitPos += n; + } + size_t get(const unsigned n) { + size_t v = peek(n); + skip(n); + return v; + } + size_t getReverse(const unsigned n) { + return bitReverse(get(n), n); + } + void skipToByte() { + skip(_bitsRemaining & 7); + } + bool checkLastBitsOfByteAreZero() { + return peek(_bitsRemaining & 7) == 0; + } + void fastFill(const unsigned n) { + if (_bitsRemaining < n) { + _fill(); + } + } + size_t fastPeek(const unsigned n) { + return _bits & ((1 << n) - 1); + } + size_t fastGet(const unsigned n) { + size_t v = fastPeek(n); + skip(n); + return v; + } + size_t copyBytesTo(OutputStream& output, const size_t len); + +private: + void _fillBytes(); + void _fill(); + + enum { BUF_SIZE = 1024, PRE_BUF_EXTRA = 16, BITS = sizeof(size_t)*8 }; + + InputStream& _input; + unsigned char _buffer[PRE_BUF_EXTRA + BUF_SIZE]; + unsigned _bufPos, _bufSize, _bufFastLimit; + bool _eof; + size_t _bits; + unsigned _bitsRemaining; + size_t _totalBitPos; +}; + +class BitOutputStream { +public: + BitOutputStream(OutputStream&); + + void put(const size_t value, const unsigned n) { + if (_bitPos + n > BITS) { + _flush(); + } + _bits |= (value & ((1 << n) - 1)) << _bitPos; + _bitPos += n; + } + void putReverse(const size_t value, const unsigned n) { + put(bitReverse(value, n), n); + } + void fillByte() { + _bitPos = (_bitPos + 7) & ~7; + } + void flush(); + unsigned bitPos() const { + return _bitPos; + } + +private: + void _flush(); + + enum { + BUF_SIZE = 1024, BUF_EXTRA = 64, BITS = sizeof(size_t) * 8 + }; + + OutputStream& _output; + unsigned char _buffer[BUF_SIZE + BUF_EXTRA]; + unsigned _bufPos; + size_t _bits; + unsigned _bitPos; +}; + +#endif /* BITSTREAM_H */ diff --git a/contrib/preflate/support/const_division.cpp b/contrib/preflate/support/const_division.cpp new file mode 100644 index 0000000..52b9679 --- /dev/null +++ b/contrib/preflate/support/const_division.cpp @@ -0,0 +1,114 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "bit_helper.h" +#include "const_division.h" + +// Based on "N-Bit Unsigned Division Via N-Bit Multiply-Add" +// by Robison + +template +udivider_t build_udivider(const typename divider_uint_t::type d) { + typedef typename divider_uint_t::type T1; + typedef typename divider_uint_t::type T2; + udivider_t result; + result.shift = bitLength(d) - 1; + if ((d & (d - 1)) == 0) { + result.magic1 = result.magic2 = ~(T2)0; + } else { + T2 shm = 1 << result.shift; + T2 t = (((T1)shm) << N) / d; + T2 r = t * d + d; + if (r <= shm) { + result.magic1 = t + 1; + result.magic2 = 0; + } else { + result.magic1 = t; + result.magic2 = t; + } + } + return result; +} + +udivider_t<16> build_udivider_16(const uint16_t d) { + return build_udivider<16>(d); +} +udivider_t<32> build_udivider_32(const uint32_t d) { + return build_udivider<32>(d); +} + +template +ucdivider_t build_ucdivider(const typename divider_uint_t::type d) { + typedef typename divider_uint_t::type T1; + typedef typename divider_uint_t::type T2; + ucdivider_t result; + result.ctrl = bitLength(d) - 1; + if ((d & (d - 1)) == 0) { + result.magic = ~(T2)0; + result.ctrl |= 0x80; + } else { + T2 shm = 1 << result.ctrl; + T2 t = (((T1)shm) << N) / d; + T2 r = t * d + d; + if (r <= shm) { + result.magic = t + 1; + } else { + result.magic = t; + result.ctrl |= 0x80; + } + } + return result; +} + +ucdivider_t<16> build_ucdivider_16(const uint16_t d) { + return build_ucdivider<16>(d); +} +ucdivider_t<32> build_ucdivider_32(const uint32_t d) { + return build_ucdivider<32>(d); +} + +template +sdivider_t build_sdivider(const typename divider_int_t::type d_) { + sdivider_t result; + udivider_t uresult = build_udivider(d_ < 0 ? -d_ : d_); + result.magic1 = uresult.magic1; + result.magic2 = uresult.magic2; + result.shift = uresult.shift; + result.sign = d_ < 0 ? -1 : 0; + return result; +} +sdivider_t<16> build_sdivider_16(const int16_t d) { + return build_sdivider<16>(d); +} +sdivider_t<32> build_sdivider_32(const int32_t d) { + return build_sdivider<32>(d); +} + +template +scdivider_t build_scdivider(const typename divider_int_t::type d_) { + scdivider_t result; + ucdivider_t uresult = build_ucdivider(d_ < 0 ? -d_ : d_); + result.magic = uresult.magic; + result.ctrl = uresult.ctrl; + if (d_ < 0) { + result.ctrl |= 0x40; + } + return result; +} +scdivider_t<16> build_scdivider_16(const int16_t d) { + return build_scdivider<16>(d); +} +scdivider_t<32> build_scdivider_32(const int32_t d) { + return build_scdivider<32>(d); +} diff --git a/contrib/preflate/support/const_division.h b/contrib/preflate/support/const_division.h new file mode 100644 index 0000000..0d197db --- /dev/null +++ b/contrib/preflate/support/const_division.h @@ -0,0 +1,170 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef CONST_DIVISION_H +#define CONST_DIVISION_H + +#include + +template struct divider_int_t; +template struct divider_uint_t; +template <> struct divider_int_t<16> { + typedef int16_t type; +}; +template <> struct divider_int_t<32> { + typedef int32_t type; +}; +template <> struct divider_uint_t<16> { + typedef uint16_t type; +}; +template <> struct divider_uint_t<32> { + typedef uint32_t type; +}; +template <> struct divider_uint_t<64> { + typedef uint64_t type; +}; + + +template +struct udivider_t { + typename divider_uint_t::type magic1; // factor + typename divider_uint_t::type magic2; // addend + uint8_t shift; +}; + +template +struct ucdivider_t { + typename divider_uint_t::type magic; // factor/addend + uint8_t ctrl; // bits 0..3/4/5 - shift, bit 7 - add required +}; + +// If it wasn't for +/-1, the signed dividers wouldn't +// need the add-term (magic2), as they could just +// use a factor (magic1) with one more bit precision. +template +struct sdivider_t { + typename divider_uint_t::type magic1; // factor + typename divider_uint_t::type magic2; // addend + uint8_t shift; + int8_t sign; // -1 if negative, 0 otherwise +}; + +template +struct scdivider_t { + typename divider_uint_t::type magic; + uint8_t ctrl; // bits 0..3/4/5 - shift, bit 6 - negative, bit 7 - add required +}; + +udivider_t<16> build_udivider_16(const uint16_t d); +udivider_t<32> build_udivider_32(const uint32_t d); + +ucdivider_t<16> build_ucdivider_16(const uint16_t d); +ucdivider_t<32> build_ucdivider_32(const uint32_t d); + +sdivider_t<16> build_sdivider_16(const int16_t d); +sdivider_t<32> build_sdivider_32(const int32_t d); + +scdivider_t<16> build_scdivider_16(const int16_t d); +scdivider_t<32> build_scdivider_32(const int32_t d); + +template +inline typename divider_uint_t::type +divide_template(const typename divider_uint_t::type dividend, + const udivider_t& divisor) { + typedef typename divider_uint_t::type T1; + typedef typename divider_uint_t::type T2; + T1 t = ((T1)dividend) * divisor.magic1 + divisor.magic2; + T2 u = (T2)(t >> N2); + return u >> divisor.shift; +} +template +inline typename divider_uint_t::type +divide_template(const typename divider_uint_t::type dividend, + const ucdivider_t& divisor) { + typedef typename divider_uint_t::type T1; + typedef typename divider_uint_t::type T2; + T1 t = ((T1)dividend) * divisor.magic + + (divisor.ctrl & 0x80 ? divisor.magic : 0); + T2 u = (T2)(t >> N2); + return u >> (divisor.ctrl & (N2 - 1)); +} +template +inline typename divider_int_t::type +divide_template(const typename divider_int_t::type dividend, + const sdivider_t& divisor) { + typedef typename divider_uint_t::type T1; + typedef typename divider_uint_t::type T2; + T2 s = dividend < 0 ? -1 : 0; + T1 t = ((T1)(T2)((dividend ^ s) - s)) * divisor.magic1 + + divisor.magic2; + T2 u = (T2)(t >> N2) >> divisor.shift; + s ^= divisor.sign; + return (u ^ s) - s; +} +template +inline typename divider_int_t::type +divide_template(const typename divider_int_t::type dividend, + const scdivider_t& divisor) { + typedef typename divider_uint_t::type T1; + typedef typename divider_uint_t::type T2; + + T2 s = dividend < 0 ? -1 : 0; + T1 t = ((T1)(T2)((dividend ^ s) - s)) * divisor.magic + + (divisor.ctrl & 0x80 ? divisor.magic : 0); + T2 u = (T2)(t >> N2) >> (divisor.ctrl & (N2 - 1)); + s ^= (divisor.ctrl & 0x40 ? -1 : 0); + return (u ^ s) - s; +} + +inline uint16_t divide(const uint16_t dividend, const udivider_t<16>& divisor) { + return divide_template<16, 16>(dividend, divisor); +} +inline uint32_t divide(const uint32_t dividend, const udivider_t<16>& divisor) { + return divide_template<32, 16>(dividend, divisor); +} +inline uint32_t divide(const uint32_t dividend, const udivider_t<32>& divisor) { + return divide_template<32, 32>(dividend, divisor); +} + +inline uint16_t divide(const uint16_t dividend, const ucdivider_t<16>& divisor) { + return divide_template<16, 16>(dividend, divisor); +} +inline uint32_t divide(const uint32_t dividend, const ucdivider_t<16>& divisor) { + return divide_template<32, 16>(dividend, divisor); +} +inline uint32_t divide(const uint32_t dividend, const ucdivider_t<32>& divisor) { + return divide_template<32, 32>(dividend, divisor); +} + +inline int16_t divide(const int16_t dividend, const sdivider_t<16>& divisor) { + return divide_template<16, 16>(dividend, divisor); +} +inline int32_t divide(const int32_t dividend, const sdivider_t<16>& divisor) { + return divide_template<32, 16>(dividend, divisor); +} +inline int32_t divide(const int32_t dividend, const sdivider_t<32>& divisor) { + return divide_template<32, 32>(dividend, divisor); +} + +inline int16_t divide(const int16_t dividend, const scdivider_t<16>& divisor) { + return divide_template<16, 16>(dividend, divisor); +} +inline int32_t divide(const int32_t dividend, const scdivider_t<16>& divisor) { + return divide_template<32, 16>(dividend, divisor); +} +inline int32_t divide(const int32_t dividend, const scdivider_t<32>& divisor) { + return divide_template<32, 32>(dividend, divisor); +} + +#endif /* CONST_DIVISION_H */ diff --git a/contrib/preflate/support/filestream.cpp b/contrib/preflate/support/filestream.cpp new file mode 100644 index 0000000..fc65eee --- /dev/null +++ b/contrib/preflate/support/filestream.cpp @@ -0,0 +1,40 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include +#include "filestream.h" + +FileStream::FileStream(FILE* f) : _f(f) {} + +bool FileStream::eof() const { + return feof(_f); +} +size_t FileStream::read(unsigned char* buffer, const size_t size) { + return fread(buffer, 1, size, _f); +} + +size_t FileStream::write(const unsigned char* buffer, const size_t size) { + return fwrite(buffer, 1, size, _f); +} + +uint64_t FileStream::tell() const { + return _ftelli64(_f); +} +uint64_t FileStream::seek(const uint64_t newPos) { + uint64_t oldPos = tell(); + _fseeki64(_f, newPos, SEEK_SET); + return oldPos; +} diff --git a/contrib/preflate/support/filestream.h b/contrib/preflate/support/filestream.h new file mode 100644 index 0000000..d12d0d2 --- /dev/null +++ b/contrib/preflate/support/filestream.h @@ -0,0 +1,38 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef FILESTREAM_H +#define FILESTREAM_H + +#include +#include +#include "stream.h" + +class FileStream : public SeekableInputOutputStream { +public: + FileStream(FILE* f); + + virtual bool eof() const; + virtual size_t read(unsigned char* buffer, const size_t size); + + virtual size_t write(const unsigned char* buffer, const size_t size); + + virtual uint64_t tell() const; + virtual uint64_t seek(const uint64_t newPos); + +private: + FILE* _f; +}; + +#endif /* FILESTREAM_H */ diff --git a/contrib/preflate/support/huffman_decoder.cpp b/contrib/preflate/support/huffman_decoder.cpp new file mode 100644 index 0000000..47dbaec --- /dev/null +++ b/contrib/preflate/support/huffman_decoder.cpp @@ -0,0 +1,110 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "huffman_decoder.h" +#include "huffman_helper.h" +#include "bit_helper.h" + +HuffmanDecoder::HuffmanDecoder( + const unsigned char* symbolBitLengths, + const size_t symbolCount, + const bool disableZeroBitSymbols, + const unsigned char maxBitsPerTable +) : _error(false) { + if (!_constructTables(symbolBitLengths, symbolCount, disableZeroBitSymbols, maxBitsPerTable)) { + _constructErrorTable(); + } +} + +size_t HuffmanDecoder::_decodeDeeper( + BitInputStream& bis, + const size_t tableId_ +) const { + bis.skip(_table0.peekBits); + size_t tableId = tableId_; + do { + const Table* table = &_tables[tableId]; + size_t v = bis.peek(table->peekBits); + signed short w = table->lookup[v]; + if (w >= 0) { + bis.skip(w & 0xf); + return w >> 4; + } + bis.skip(table->peekBits); + tableId = ~w; + } while (true); +} +bool HuffmanDecoder::_constructTables( + const unsigned char* symbolBitLengths, + const size_t symbolCount, + const bool disableZeroBitSymbols, + const unsigned char maxBitsPerTable +) { + if (maxBitsPerTable < 1 || maxBitsPerTable > 15) { + return false; + } + unsigned nextCode[HuffmanHelper::MAX_BL + 2]; + unsigned char minLength, maxLength; + if (!HuffmanHelper::countSymbols(nextCode, minLength, maxLength, + symbolBitLengths, symbolCount, + disableZeroBitSymbols)) { + return false; + } + + _table0.peekBits = std::min((unsigned char)(maxLength - 1), maxBitsPerTable); + _table0.lookup.resize(1 << _table0.peekBits); + std::fill(_table0.lookup.begin(), _table0.lookup.end(), 0); + + unsigned char minL = disableZeroBitSymbols ? 2 : 1; + + for (unsigned i = 0; i < symbolCount; ++i) { + unsigned char l = (unsigned char)(symbolBitLengths[i] + 1); + if (l < minL) { + continue; + } + unsigned char k = l - 1, maxK = maxLength - 1; + unsigned code = bitReverse(nextCode[l]++, k); + Table* t = &_table0; + while (k > t->peekBits) { + k -= t->peekBits; + maxK -= t->peekBits; + unsigned subbits = code & ((1 << t->peekBits) - 1); + code >>= t->peekBits; + signed short v = t->lookup[subbits]; + if (v >= 0) { + unsigned newTableId = _tables.size(); + t->lookup[subbits] = ~newTableId; + _tables.push_back(Table()); + t = &_tables[newTableId]; + t->peekBits = std::min(maxK, maxBitsPerTable); + t->lookup.resize(1 << t->peekBits); + std::fill(t->lookup.begin(), t->lookup.end(), 0); + } else { + t = &_tables[~v]; + } + } + do { + t->lookup[code] = (i << 4) | k; + code += 1 << k; + } while (code < t->lookup.size()); + } + return true; +} +void HuffmanDecoder::_constructErrorTable() { + _error = true; + _table0.peekBits = 0; + _table0.lookup.resize(1); + _table0.lookup[0] = 0; +} diff --git a/contrib/preflate/support/huffman_decoder.h b/contrib/preflate/support/huffman_decoder.h new file mode 100644 index 0000000..264ebe6 --- /dev/null +++ b/contrib/preflate/support/huffman_decoder.h @@ -0,0 +1,62 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef HUFFMAN_DECODER_H +#define HUFFMAN_DECODER_H + +#include +#include "bitstream.h" + +// Huffman decoder +class HuffmanDecoder { +public: + HuffmanDecoder(const unsigned char* symbolBitLengths, + const size_t symbolCount, + const bool disableZeroBitSymbols, + const unsigned char maxBitsPerTable); + + bool error() const { + return _error; + } + + size_t decode(BitInputStream& bis) const { + size_t v = bis.peek(_table0.peekBits); + signed short w = _table0.lookup[v]; + if (w >= 0) { + bis.skip(w & 0xf); + return w >> 4; + } + return _decodeDeeper(bis, ~w); + } + +private: + size_t _decodeDeeper(BitInputStream& bis, const size_t tableId) const; + bool _constructTables(const unsigned char* symbolBitLengths, + const size_t symbolCount, + const bool disableZeroBitSymbols, + const unsigned char maxBitsPerTable); + void _constructErrorTable(); + +private: + struct Table { + unsigned char peekBits; + std::vector lookup; + }; + + Table _table0; + std::vector _tables; + bool _error; +}; + +#endif /* HUFFMAN_DECODER_H */ diff --git a/contrib/preflate/support/huffman_encoder.cpp b/contrib/preflate/support/huffman_encoder.cpp new file mode 100644 index 0000000..caec63c --- /dev/null +++ b/contrib/preflate/support/huffman_encoder.cpp @@ -0,0 +1,64 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "huffman_encoder.h" +#include "huffman_helper.h" +#include "bit_helper.h" + +HuffmanEncoder::HuffmanEncoder( + const unsigned char* symbolBitLengths, + const unsigned symbolCount, + const bool disableZeroBitSymbols +) : _error(false) { + if (!_constructTables(symbolBitLengths, symbolCount, disableZeroBitSymbols)) { + _constructErrorTable(symbolCount); + } +} + +bool HuffmanEncoder::_constructTables( + const unsigned char* symbolBitLengths, + const unsigned symbolCount, + const bool disableZeroBitSymbols +) { + unsigned nextCode[HuffmanHelper::MAX_BL + 2]; + unsigned char minLength, maxLength; + if (!HuffmanHelper::countSymbols(nextCode, minLength, maxLength, + symbolBitLengths, symbolCount, + disableZeroBitSymbols)) { + return false; + } + + unsigned char minL = disableZeroBitSymbols ? 2 : 1; + + _lookup.resize(symbolCount); + for (unsigned i = 0; i < symbolCount; ++i) { + unsigned char l = (unsigned char)(symbolBitLengths[i] + 1); + if (l < minL) { + _lookup[i] = 0; + continue; + } + unsigned char k = l - 1; + unsigned code = bitReverse(nextCode[l]++, k); + _lookup[i] = (code << 5) | k; + } + return true; +} +void HuffmanEncoder::_constructErrorTable( + const unsigned symbolCount +) { + _error = true; + _lookup.resize(symbolCount); + std::fill(_lookup.begin(), _lookup.end(), 0); +} diff --git a/contrib/preflate/support/huffman_encoder.h b/contrib/preflate/support/huffman_encoder.h new file mode 100644 index 0000000..a499846 --- /dev/null +++ b/contrib/preflate/support/huffman_encoder.h @@ -0,0 +1,48 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef HUFFMAN_ENCODER_H +#define HUFFMAN_ENCODER_H + +#include +#include "bitstream.h" + +// Huffman decoder +class HuffmanEncoder { +public: + HuffmanEncoder(const unsigned char* symbolBitLengths, + const unsigned symbolCount, + const bool disableZeroBitSymbols); + + bool error() const { + return _error; + } + + void encode(BitOutputStream& bos, const unsigned symbol) const { + unsigned v = _lookup[symbol]; + bos.put(v >> 5, v & 0x1f); + } + +private: + bool _constructTables(const unsigned char* symbolBitLengths, + const unsigned symbolCount, + const bool disableZeroBitSymbols); + void _constructErrorTable(const unsigned symbolCount); + +private: + std::vector _lookup; + bool _error; +}; + +#endif /* HUFFMAN_ENCODER_H */ diff --git a/contrib/preflate/support/huffman_helper.cpp b/contrib/preflate/support/huffman_helper.cpp new file mode 100644 index 0000000..b3638f5 --- /dev/null +++ b/contrib/preflate/support/huffman_helper.cpp @@ -0,0 +1,74 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "huffman_helper.h" +#include "bit_helper.h" + +bool HuffmanHelper::countSymbols( + unsigned(&nextCode)[MAX_BL + 2], + unsigned char& minLength, + unsigned char& maxLength, + const unsigned char* symbolBitLengths, + const unsigned symbolCount, + const bool disableZeroBitSymbols +) { + if (symbolCount < 1 || symbolCount >= 1024) { + return false; + } + unsigned short blCount[MAX_BL + 2]; + + // Count symbol frequencies + memset(blCount, 0, sizeof(blCount)); + for (unsigned i = 0; i < symbolCount; ++i) { + unsigned char l = (unsigned char)(symbolBitLengths[i] + 1); + if (l > MAX_BL + 1) { + return false; + } + blCount[l]++; + } + for (minLength = 1; minLength <= MAX_BL + 1; ++minLength) { + if (blCount[minLength]) { + break; + } + } + for (maxLength = MAX_BL + 1; maxLength >= minLength; --maxLength) { + if (blCount[maxLength]) { + break; + } + } + if (minLength > maxLength) { + return false; + } + // Remove deleted symbols + blCount[0] = 0; + if (disableZeroBitSymbols) { + blCount[1] = 0; + } + + // Calculate start codes + unsigned code = 0; + for (unsigned i = minLength; i <= maxLength; ++i) { + code = (code + blCount[i - 1]) << 1; + nextCode[i] = code; + } + + if (minLength == maxLength && blCount[maxLength] == 1) { + return true; + } + + // Check that we don't have holes + return nextCode[maxLength] + blCount[maxLength] == (unsigned)(1 << (maxLength - 1)); +} diff --git a/contrib/preflate/support/huffman_helper.h b/contrib/preflate/support/huffman_helper.h new file mode 100644 index 0000000..39d33f7 --- /dev/null +++ b/contrib/preflate/support/huffman_helper.h @@ -0,0 +1,34 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef HUFFMAN_HELPER_H +#define HUFFMAN_HELPER_H + +#include + +// Huffman decoder +class HuffmanHelper { +public: + enum { + MAX_BL = 25 + }; + static bool countSymbols(unsigned(&nextCode)[MAX_BL + 2], + unsigned char& minLength, + unsigned char& maxLength, + const unsigned char* symbolBitLengths, + const unsigned symbolCount, + const bool disableZeroBitSymbols); +}; + +#endif /* HUFFMAN_HELPER_H */ diff --git a/contrib/preflate/support/memstream.cpp b/contrib/preflate/support/memstream.cpp new file mode 100644 index 0000000..9ef158f --- /dev/null +++ b/contrib/preflate/support/memstream.cpp @@ -0,0 +1,51 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include +#include "memstream.h" + +MemStream::MemStream() : _pos(0) {} +MemStream::MemStream(const std::vector& content) + : _data(content) + , _pos(0) {} + +bool MemStream::eof() const { + return _pos == _data.size(); +} +size_t MemStream::read(unsigned char* buffer, const size_t size) { + size_t toCopy = std::min(size, _data.size() - _pos); + memcpy(buffer, _data.data() + _pos, toCopy); + _pos += toCopy; + return toCopy; +} + +size_t MemStream::write(const unsigned char* buffer, const size_t size) { + size_t remaining = _data.size() - _pos; + if (size > remaining) { + _data.resize(_pos + size); + } + memcpy(_data.data() + _pos, buffer, size); + _pos += size; + return size; +} + +uint64_t MemStream::tell() const { + return _pos; +} +uint64_t MemStream::seek(const uint64_t newPos) { + size_t oldPos = _pos; + _pos = std::min(newPos, (uint64_t)_data.size()); + return oldPos; +} diff --git a/contrib/preflate/support/memstream.h b/contrib/preflate/support/memstream.h new file mode 100644 index 0000000..0e2ed87 --- /dev/null +++ b/contrib/preflate/support/memstream.h @@ -0,0 +1,47 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef MEMSTREAM_H +#define MEMSTREAM_H + +#include +#include +#include "stream.h" + +class MemStream : public SeekableInputOutputStream { +public: + MemStream(); + MemStream(const std::vector& content); + + virtual bool eof() const; + virtual size_t read(unsigned char* buffer, const size_t size); + + virtual size_t write(const unsigned char* buffer, const size_t size); + + virtual uint64_t tell() const; + virtual uint64_t seek(const uint64_t newPos); + + const std::vector& data() const { + return _data; + } + std::vector extractData() { + return std::move(_data); + } + +private: + std::vector _data; + size_t _pos; +}; + +#endif /* MEMSTREAM_H */ diff --git a/contrib/preflate/support/outputcachestream.cpp b/contrib/preflate/support/outputcachestream.cpp new file mode 100644 index 0000000..d3d2996 --- /dev/null +++ b/contrib/preflate/support/outputcachestream.cpp @@ -0,0 +1,29 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "outputcachestream.h" + +OutputCacheStream::OutputCacheStream(OutputStream& os) + : _os(os) + , _cacheStartPos(0) {} +OutputCacheStream::~OutputCacheStream() { +} + +void OutputCacheStream::flushUpTo(const uint64_t newStartPos) { + size_t toWrite = std::min(newStartPos - _cacheStartPos, (uint64_t)_cache.size()); + size_t written = _os.write(_cache.data(), toWrite); + _cacheStartPos += written; + _cache.erase(_cache.begin(), _cache.begin() + written); +} diff --git a/contrib/preflate/support/outputcachestream.h b/contrib/preflate/support/outputcachestream.h new file mode 100644 index 0000000..d95c236 --- /dev/null +++ b/contrib/preflate/support/outputcachestream.h @@ -0,0 +1,67 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef OUTPUTCACHESTREAM_H +#define OUTPUTCACHESTREAM_H + +#include +#include +#include "stream.h" + +class OutputCacheStream : public OutputStream { +public: + OutputCacheStream(OutputStream& os); + ~OutputCacheStream(); + + size_t write(const unsigned char* buffer, const size_t size) { +/* if (size == 1) { + _cache.push_back(*buffer); + return 1; + }*/ + _cache.insert(_cache.end(), buffer, buffer + size); + return size; + } + void reserve(const size_t len) { + size_t cap = _cache.capacity(); + if (_cache.size() + len > cap) { + _cache.reserve(cap + std::max(cap >> 1, len)); + } + } + void flush() { + flushUpTo(cacheEndPos()); + } + void flushUpTo(const uint64_t newStartPos); + uint64_t cacheStartPos() const { + return _cacheStartPos; + } + uint64_t cacheEndPos() const { + return _cacheStartPos + _cache.size(); + } + const unsigned char* cacheData(const uint64_t pos) const { + return _cache.data() + (ptrdiff_t)(pos - _cacheStartPos); + } + const unsigned char* cacheEnd() const { + return _cache.data() + _cache.size(); + } + const size_t cacheSize() const { + return _cache.size(); + } + +private: + OutputStream& _os; + std::vector _cache; + uint64_t _cacheStartPos; +}; + +#endif /* OUTPUTCACHESTREAM_H */ diff --git a/contrib/preflate/support/stream.h b/contrib/preflate/support/stream.h new file mode 100644 index 0000000..f41f0c6 --- /dev/null +++ b/contrib/preflate/support/stream.h @@ -0,0 +1,45 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef STREAM_H +#define STREAM_H + +#include + +class InputStream { +public: + virtual bool eof() const = 0; + virtual size_t read(unsigned char* buffer, const size_t size) = 0; +}; + +class OutputStream { +public: + virtual size_t write(const unsigned char* buffer, const size_t size) = 0; +}; + +class SeekableStream { +public: + virtual uint64_t tell() const = 0; + virtual uint64_t seek(const uint64_t newPos) = 0; +}; + +class SeekableInputStream + : public InputStream + , public SeekableStream {}; +class SeekableInputOutputStream + : public SeekableInputStream + , public OutputStream {}; + + +#endif /* STREAM_H */ diff --git a/contrib/preflate/support/support_tests.cpp b/contrib/preflate/support/support_tests.cpp new file mode 100644 index 0000000..b633bae --- /dev/null +++ b/contrib/preflate/support/support_tests.cpp @@ -0,0 +1,180 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include +#include "array_helper.h" +#include "bit_helper.h" +#include "bitstream.h" +#include "const_division.h" +#include "huffman_decoder.h" +#include "huffman_encoder.h" +#include "huffman_helper.h" +#include "memstream.h" +#include "outputcachestream.h" +#include "stream.h" + +bool support_self_tests() { + unsigned arr[] = {1,2,3,4,5}; + if (sumArray(arr) != 15 + || sumArray(arr, sizeof(arr) / sizeof(arr[0])) != 15) { + printf("sumArray failed\n"); + return false; + } + if (bitLength(0) != 0 + || bitLength(15) != 4 + || bitLength(0xffffffff) != 32) { + printf("bitLength failed\n"); + return false; + } + if (bitReverse(1, 3) != 4 + || bitReverse(0x12345678, 32) != 0x1e6a2c48 + || bitReverse(0xfedcba90, 32) != 0x095d3b7f) { + printf("bitReverse failed\n"); + return false; + } + + MemStream mem; + mem.write((const uint8_t*)"Hello", 5); + if (mem.tell() != 5 || !mem.eof()) { + printf("MemStream/1 failed\n"); + return false; + } + mem.write((const uint8_t*)"!", 1); + uint8_t tmp[5], tmp2[2]; + if (mem.read(tmp, 5) != 0) { + printf("MemStream/2 failed\n"); + return false; + } + if (mem.seek(0) != 6) { + printf("MemStream/3 failed\n"); + return false; + } + if (mem.tell() != 0) { + printf("MemStream/4 failed\n"); + return false; + } + if (mem.read(tmp, 5) != 5 || tmp[0] != 'H' || tmp[4] != 'o') { + printf("MemStream/5 failed\n"); + return false; + } + if (mem.read(tmp2, 2) != 1 || tmp2[0] != '!') { + printf("MemStream/6 failed\n"); + return false; + } + if (!mem.eof()) { + printf("MemStream/7 failed\n"); + return false; + } + + mem.seek(0); + { + BitOutputStream bos(mem); + for (unsigned i = 0; i <= HuffmanHelper::MAX_BL; ++i) { + bos.put(i, i); + } + bos.flush(); + } + mem.seek(0); + { + BitInputStream bis(mem); + for (unsigned i = 0; i <= HuffmanHelper::MAX_BL; ++i) { + if (bis.get(i) != i) { + printf("BitStreams failed\n"); + return false; + } + } + } + + unsigned char lengths[] = { + 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16, + 17,18,19,20,21,22,23,24,25,25 + }; + unsigned count = sizeof(lengths) / sizeof(lengths[0]); + HuffmanEncoder henc(lengths, count, false); + HuffmanDecoder hdec(lengths, count, false, 7); + if (henc.error() || hdec.error()) { + printf("HuffmanEncoder failed\n"); + return false; + } + mem.seek(0); + { + BitOutputStream bos(mem); + for (unsigned i = 0; i < count; ++i) { + henc.encode(bos, i); + } + bos.flush(); + } + mem.seek(0); + { + BitInputStream bis(mem); + for (unsigned i = 0; i < count; ++i) { + if (hdec.decode(bis) != i) { + printf("HuffmanDecoder failed\n"); + return false; + } + } + } + + uint16_t divtest16[] = {1, 3, 5, 7, 9, 11, 13, 17, 32767}; + for (int i = 0, n = sizeof(divtest16) / sizeof(divtest16[0]); i < n; ++i) { + udivider_t<16> du = build_udivider_16(divtest16[i]); + ucdivider_t<16> duc = build_ucdivider_16(divtest16[i]); + sdivider_t<16> ds = build_sdivider_16(divtest16[i]); + scdivider_t<16> dsc = build_scdivider_16(divtest16[i]); + + for (int k = 0; k < 65536; ++k) { + uint16_t c1 = divide((uint16_t)k, du); + uint16_t c2 = divide((uint16_t)k, duc); + uint16_t r = k / divtest16[i]; + if (c1 != r || c2 != r) { + printf("16bit divider/1 failed\n"); + return false; + } + + int16_t d1 = divide((int16_t)(k - 32768), ds); + int16_t d2 = divide((int16_t)(k - 32768), dsc); + int16_t s = ((int16_t)(k - 32768)) / (int16_t)divtest16[i]; + if (d1 != s || d2 != s) { + printf("16bit divider/2 failed\n"); + return false; + } + } + } + uint32_t divtest32[] = {1, 3, 5, 7, 9, 11, 13, 17, 0x7fff, 0x7fffffff}; + for (int i = 0, n = sizeof(divtest32) / sizeof(divtest32[0]); i < n; ++i) { + udivider_t<32> du = build_udivider_32(divtest32[i]); + ucdivider_t<32> duc = build_ucdivider_32(divtest32[i]); + sdivider_t<32> ds = build_sdivider_32(divtest32[i]); + scdivider_t<32> dsc = build_scdivider_32(divtest32[i]); + + for (int k = 0; k < 65536; ++k) { + uint32_t c1 = divide(((uint32_t)k)* 65536, du); + uint32_t c2 = divide(((uint32_t)k) * 65536, duc); + uint32_t r = (((uint32_t)k) * 65536) / divtest32[i]; + if (c1 != r || c2 != r) { + printf("32bit divider/1 failed\n"); + return false; + } + + int32_t d1 = divide((int32_t)(k - 32768) * 65536, ds); + int32_t d2 = divide((int32_t)(k - 32768) * 65536, dsc); + int32_t s = ((int32_t)(k - 32768)) * 65536 / (int32_t)divtest32[i]; + if (d1 != s || d2 != s) { + printf("32bit divider/2 failed\n"); + return false; + } + } + } + return true; +} diff --git a/contrib/preflate/support/support_tests.h b/contrib/preflate/support/support_tests.h new file mode 100644 index 0000000..5ce4efd --- /dev/null +++ b/contrib/preflate/support/support_tests.h @@ -0,0 +1,20 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef SUPPORT_TESTS_H +#define SUPPORT_TESTS_H + +bool support_self_tests(); + +#endif /* SUPPORT_TESTS_H */ diff --git a/contrib/preflate/support/task_pool.cpp b/contrib/preflate/support/task_pool.cpp new file mode 100644 index 0000000..cc56aea --- /dev/null +++ b/contrib/preflate/support/task_pool.cpp @@ -0,0 +1,15 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "task_pool.h" diff --git a/contrib/preflate/support/task_pool.h b/contrib/preflate/support/task_pool.h new file mode 100644 index 0000000..4265552 --- /dev/null +++ b/contrib/preflate/support/task_pool.h @@ -0,0 +1,48 @@ +/* Copyright 2018 Dirk Steinke + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#ifndef TASK_POOL_H +#define TASK_POOL_H + +#include +#include + +class Task { +public: + virtual bool execute() = 0; +}; +class FunctionTask : public Task { +public: + FunctionTask(std::function && fun) + : _fun(fun) { + } + + bool execute() { + return _fun(); + } + +private: + std::function _fun; +}; + +class TaskPool { +public: + TaskPool(); + ~TaskPool(); + + void addTask(std::unique_ptr&& task); + void addTask(std::function && fun); +}; + +#endif /* TASK_POOL_H */ diff --git a/precomp.cpp b/precomp.cpp index 58c77ba..991e978 100644 --- a/precomp.cpp +++ b/precomp.cpp @@ -19,9 +19,9 @@ // version information #define V_MAJOR 0 #define V_MINOR 4 -#define V_MINOR2 7 +#define V_MINOR2 128 //#define V_STATE "ALPHA" -#define V_STATE "DEVELOPMENT" +#define V_STATE "EXPERIMENTAL (w/ preflate support)" #define V_MSG "USE FOR TESTING ONLY" //#define V_MSG "USE AT YOUR OWN RISK!" #ifdef UNIX @@ -95,6 +95,7 @@ using namespace std; #include "contrib/packjpg/precomp_jpg.h" #include "contrib/packmp3/precomp_mp3.h" #include "contrib/zlib/zlib.h" +#include "contrib/preflate/preflate.h" #define CHUNK 262144 // 256 KB buffersize #define DIV3CHUNK 262143 // DIV3CHUNK is a bit smaller/larger than CHUNK, so that DIV3CHUNK mod 3 = 0 @@ -306,6 +307,22 @@ bool use_swf = true; bool use_base64 = true; bool use_bzip2 = true; +enum { + D_PDF = 0, + D_ZIP = 1, + D_GZIP = 2, + D_PNG = 3, + D_MULTIPNG = 4, + D_GIF = 5, + D_JPG = 6, + D_SWF = 7, + D_BASE64 = 8, + D_BZIP2 = 9, + D_MP3 = 10, + D_RAW = 255, + D_BRUTE = 254, +}; + // Precomp DLL things #ifdef PRECOMPDLL @@ -519,7 +536,8 @@ int init(int argc, char* argv[]) { printf("Precomp v%i.%i.%i %s %s - %s version",V_MAJOR,V_MINOR,V_MINOR2,V_OS,V_BIT,V_STATE); } printf(" - %s\n",V_MSG); - printf("Free for non-commercial use - Copyright 2006-2018 by Christian Schneider\n\n"); + printf("Free for non-commercial use - Copyright 2006-2018 by Christian Schneider\n"); + printf("- experimental preflate support - Copyright 2018 by Dirk Steinke\n\n"); // init compression and memory level count bool use_zlib_level[81]; @@ -2418,51 +2436,6 @@ void denit() { delete[] best_penalty_bytes; } -int def(FILE *source, FILE *dest, int level, int windowbits, int memlevel) { - int ret, flush; - unsigned have; - z_stream strm; - - /* allocate deflate state */ - strm.zalloc = Z_NULL; - strm.zfree = Z_NULL; - strm.opaque = Z_NULL; - ret = deflateInit2(&strm, level, Z_DEFLATED, windowbits, memlevel, Z_DEFAULT_STRATEGY); - if (ret != Z_OK) - return ret; - - /* compress until end of file */ - do { - print_work_sign(true); - - strm.avail_in = own_fread(in, 1, CHUNK, source); - if (ferror(source)) { - (void)deflateEnd(&strm); - return Z_ERRNO; - } - flush = feof(source) ? Z_FINISH : Z_NO_FLUSH; - strm.next_in = in; - - do { - strm.avail_out = CHUNK; - strm.next_out = out; - - ret = deflate(&strm, flush); - - have = CHUNK - strm.avail_out; - - if (own_fwrite(out, 1, have, dest) != have || ferror(dest)) { - (void)deflateEnd(&strm); - return Z_ERRNO; - } - } while (strm.avail_out == 0); - - } while (flush != Z_FINISH); - - (void)deflateEnd(&strm); - return Z_OK; -} - // Brute mode detects a bit less than intense mode to avoid false positives // and slowdowns, so both can be active. Also, both of them can have a level // limit, so two helper functions make things easier to handle. @@ -2491,88 +2464,6 @@ void copy_penalty_bytes(long long& rek_penalty_bytes_len, bool& use_penalty_byte } #define DEF_COMPARE_CHUNK 512 -long long def_compare(FILE *compfile, int level, int windowbits, int memlevel, long long& decompressed_bytes_used, long long decompressed_bytes_total, bool in_memory) { - int ret, flush; - unsigned have; - z_stream strm; - long long identical_bytes_compare = 0; - - long long comp_pos = 0; - decompressed_bytes_used = 0; - - /* allocate deflate state */ - strm.zalloc = Z_NULL; - strm.zfree = Z_NULL; - strm.opaque = Z_NULL; - ret = deflateInit2(&strm, level, Z_DEFLATED, windowbits, memlevel, Z_DEFAULT_STRATEGY); - if (ret != Z_OK) - return -1; - - long long total_same_byte_count = 0; - long long total_same_byte_count_penalty = 0; - long long rek_same_byte_count = 0; - long long rek_same_byte_count_penalty = -1; - long long rek_penalty_bytes_len = 0; - long long local_penalty_bytes_len = 0; - bool use_penalty_bytes = false; - - unsigned char* buf_ptr = decomp_io_buf; - long long buf_pos = 0; - /* compress until end of file */ - do { - print_work_sign(true); - - if (in_memory) { - strm.avail_in = min((int)(decompressed_bytes_total - buf_pos), DEF_COMPARE_CHUNK); - strm.next_in = buf_ptr + buf_pos; - buf_pos += strm.avail_in; - flush = (buf_pos >= decompressed_bytes_total) ? Z_FINISH : Z_NO_FLUSH; - } else { - strm.avail_in = own_fread(in, 1, DEF_COMPARE_CHUNK, ftempout); - if (ferror(ftempout)) { - (void)deflateEnd(&strm); - return -1; - } - flush = feof(ftempout) ? Z_FINISH : Z_NO_FLUSH; - strm.next_in = in; - } - decompressed_bytes_used += strm.avail_in; - - do { - strm.avail_out = DEF_COMPARE_CHUNK; - strm.next_out = out; - - ret = deflate(&strm, flush); - - have = DEF_COMPARE_CHUNK - strm.avail_out; - - if (have > 0) { - if (compfile == fin) { - identical_bytes_compare = compare_file_mem_penalty(compfile, out, input_file_pos + comp_pos, have, total_same_byte_count, total_same_byte_count_penalty, rek_same_byte_count, rek_same_byte_count_penalty, rek_penalty_bytes_len, local_penalty_bytes_len, use_penalty_bytes); - } else { - identical_bytes_compare = compare_file_mem_penalty(compfile, out, comp_pos, have, total_same_byte_count, total_same_byte_count_penalty, rek_same_byte_count, rek_same_byte_count_penalty, rek_penalty_bytes_len, local_penalty_bytes_len, use_penalty_bytes); - } - } - - if (have > 0) { - if ((unsigned int)identical_bytes_compare < (have >> 1)) { - (void)deflateEnd(&strm); - copy_penalty_bytes(rek_penalty_bytes_len, use_penalty_bytes); - return rek_same_byte_count; - } - } - - comp_pos += have; - - } while (strm.avail_out == 0); - - } while (flush != Z_FINISH); - - (void)deflateEnd(&strm); - copy_penalty_bytes(rek_penalty_bytes_len, use_penalty_bytes); - return rek_same_byte_count; -} - long long def_compare_bzip2(FILE *source, FILE *compfile, int level, long long& decompressed_bytes_used) { int ret, flush; unsigned have; @@ -2646,65 +2537,6 @@ long long def_compare_bzip2(FILE *source, FILE *compfile, int level, long long& return rek_same_byte_count; } -int def_part(FILE *source, FILE *dest, int level, int windowbits, int memlevel, long long stream_size_in, long long stream_size_out) { - int ret, flush; - unsigned have; - z_stream strm; - - /* allocate deflate state */ - strm.zalloc = Z_NULL; - strm.zfree = Z_NULL; - strm.opaque = Z_NULL; - ret = deflateInit2(&strm, level, Z_DEFLATED, windowbits, memlevel, Z_DEFAULT_STRATEGY); - if (ret != Z_OK) - return ret; - - long long pos_in = 0; - long long pos_out = 0; - - /* compress until end of file */ - do { - if ((stream_size_in - pos_in) > CHUNK) { - print_work_sign(true); - - strm.avail_in = own_fread(in, 1, CHUNK, source); - pos_in += CHUNK; - flush = Z_NO_FLUSH; - } else { - strm.avail_in = own_fread(in, 1, stream_size_in - pos_in, source); - flush = Z_FINISH; - } - if (ferror(source)) { - (void)deflateEnd(&strm); - return Z_ERRNO; - } - strm.next_in = in; - - do { - strm.avail_out = CHUNK; - strm.next_out = out; - - ret = deflate(&strm, flush); - - have = CHUNK - strm.avail_out; - - if ((pos_out + (signed)have) > stream_size_out) { - have = stream_size_out - pos_out; - } - pos_out += have; - - if (own_fwrite(out, 1, have, dest) != have || ferror(dest)) { - (void)deflateEnd(&strm); - return Z_ERRNO; - } - } while (strm.avail_out == 0); - - } while (flush != Z_FINISH); - - (void)deflateEnd(&strm); - return Z_OK; -} - int def_part_bzip2(FILE *source, FILE *dest, int level, long long stream_size_in, long long stream_size_out) { int ret, flush; unsigned have; @@ -2796,162 +2628,6 @@ size_t fread_skip(unsigned char *ptr, size_t size, size_t count, FILE* stream) { return bytes_read; } -int def_part_skip(FILE *source, FILE *dest, int level, int windowbits, int memlevel, long long stream_size_in, long long stream_size_out, int bmp_width) { - - int ret, flush; - unsigned have; - z_stream strm; - - /* allocate deflate state */ - strm.zalloc = Z_NULL; - strm.zfree = Z_NULL; - strm.opaque = Z_NULL; - ret = deflateInit2(&strm, level, Z_DEFLATED, windowbits, memlevel, Z_DEFAULT_STRATEGY); - if (ret != Z_OK) - return ret; - - long long pos_in = 0; - long long pos_out = 0; - frs_offset = 0; - frs_skip_len = (4 - (bmp_width % 4)); - frs_line_len = bmp_width; - - /* compress until end of file */ - do { - if ((stream_size_in - pos_in) >= CHUNK) { - strm.avail_in = fread_skip(in, 1, CHUNK, source); - pos_in += strm.avail_in; - flush = Z_NO_FLUSH; - } else { - strm.avail_in = fread_skip(in, 1, stream_size_in - pos_in, source); - pos_in += strm.avail_in; - if (pos_in >= stream_size_in) { - flush = Z_FINISH; - } else { - flush = Z_NO_FLUSH; - } - } - if (ferror(source)) { - (void)deflateEnd(&strm); - return Z_ERRNO; - } - strm.next_in = in; - - do { - strm.avail_out = CHUNK; - strm.next_out = out; - - ret = deflate(&strm, flush); - - have = CHUNK - strm.avail_out; - - if ((pos_out + (signed)have) > stream_size_out) { - have = stream_size_out - pos_out; - } - pos_out += have; - - if (own_fwrite(out, 1, have, dest) != have || ferror(dest)) { - (void)deflateEnd(&strm); - return Z_ERRNO; - } - } while (strm.avail_out == 0); - - } while (flush != Z_FINISH); - - (void)deflateEnd(&strm); - return Z_OK; -} - -int inf(FILE *source, int windowbits, long long& compressed_stream_size, long long& decompressed_stream_size, bool& in_memory) { - int ret; - unsigned have; - z_stream strm; - - /* allocate inflate state */ - strm.zalloc = Z_NULL; - strm.zfree = Z_NULL; - strm.opaque = Z_NULL; - strm.avail_in = 0; - strm.next_in = Z_NULL; - ret = inflateInit2(&strm, windowbits); - if (ret != Z_OK) - return ret; - - compressed_stream_size = 0; - decompressed_stream_size = 0; - int avail_in_before; - - /* decompress until deflate stream ends or end of file */ - do { - print_work_sign(true); - - strm.avail_in = own_fread(in, 1, CHUNK, source); - avail_in_before = strm.avail_in; - - if (ferror(source)) { - (void)inflateEnd(&strm); - if (!in_memory) safe_fclose(&ftempout); - return Z_ERRNO; - } - if (strm.avail_in == 0) - break; - strm.next_in = in; - - /* run inflate() on input until output buffer not full */ - unsigned char* buf_ptr; - do { - strm.avail_out = CHUNK; - if (in_memory) { - buf_ptr = decomp_io_buf + decompressed_stream_size; - strm.next_out = buf_ptr; - } - else { - strm.next_out = out; - } - - ret = inflate(&strm, Z_NO_FLUSH); - switch (ret) { - case Z_NEED_DICT: - ret = Z_DATA_ERROR; - case Z_DATA_ERROR: - case Z_MEM_ERROR: - (void)inflateEnd(&strm); - if (!in_memory) safe_fclose(&ftempout); - return ret; - } - - compressed_stream_size += (avail_in_before - strm.avail_in); - avail_in_before = strm.avail_in; - - have = CHUNK - strm.avail_out; - - if (!in_memory) { - if (own_fwrite(out, 1, have, ftempout) != have || ferror(ftempout)) { - (void)inflateEnd(&strm); - safe_fclose(&ftempout); - return Z_ERRNO; - } - } - else { - if ((decompressed_stream_size + have + CHUNK) >= MAX_IO_BUFFER_SIZE) { - in_memory = false; - write_ftempout_if_not_present(decompressed_stream_size + have, true, true); - } - } - decompressed_stream_size += have; - - } while (strm.avail_out == 0); - - /* done when inflate() says it's done */ - } while (ret != Z_STREAM_END); - - /* clean up and return */ - (void)inflateEnd(&strm); - if (!in_memory) safe_fclose(&ftempout); - return ret == Z_STREAM_END ? Z_OK : Z_DATA_ERROR; - -} - int histogram[256]; bool check_inf_result(int cb_pos, int windowbits, bool use_brute_parameters = false) { @@ -3169,28 +2845,6 @@ int def_bzip2(FILE *source, FILE *dest, int level) { return BZ_OK; } -long long file_recompress(FILE* origfile, int compression_level, int windowbits, int memlevel, long long& decompressed_bytes_used, long long decompressed_bytes_total, bool in_memory) { - long long retval; - - if (!in_memory) { - ftempout = fopen(tempfile1,"rb"); - if (ftempout == NULL) { - error(ERR_TEMP_FILE_DISAPPEARED); - } - fseek(ftempout, 0, SEEK_SET); - } - - retval = def_compare(origfile, compression_level, windowbits, memlevel, decompressed_bytes_used, decompressed_bytes_total, in_memory); - - if (!in_memory) { - safe_fclose(&ftempout); - } - - if (retval < 0) return -1; - - return retval; -} - long long file_recompress_bzip2(FILE* origfile, int level, long long& decompressed_bytes_used, long long& decompressed_bytes_total) { long long retval; @@ -3324,7 +2978,7 @@ void end_uncompressed_data() { if (!uncompressed_data_in_work) return; - fout_fput64(uncompressed_length); + fout_fput_vlint(uncompressed_length); // fast copy of uncompressed data seek_64(fin, uncompressed_pos); @@ -3356,386 +3010,479 @@ void init_decompression_variables() { final_compression_found = false; } +struct recompress_deflate_result { + long long compressed_stream_size; + long long uncompressed_stream_size; + std::vector recon_data; + bool accepted; + bool uncompressed_in_memory; + bool zlib_perfect; + char zlib_comp_level; + char zlib_mem_level; + char zlib_window_bits; +}; + +void debug_deflate_detected(const recompress_deflate_result& rdres, const char* type) { + if (DEBUG_MODE) { + print_debug_percent(); + cout << "Possible zLib-Stream " << type << " found at position " << saved_input_file_pos << endl; + cout << "Compressed size: " << rdres.compressed_stream_size << endl; + cout << "Can be decompressed to " << rdres.uncompressed_stream_size << " bytes" << endl; + + if (rdres.accepted) { + if (rdres.zlib_perfect) { + cout << "Detect ZLIB parameters: comp level " << rdres.zlib_comp_level << ", mem level " << rdres.zlib_mem_level << ", " << rdres.zlib_window_bits << "window bits" << endl; + } else { + cout << "Non-ZLIB reconstruction data size: " << rdres.recon_data.size() << " bytes" << endl; + } + } + } +} +void debug_deflate_reconstruct(const recompress_deflate_result& rdres, const char* type, + const unsigned hdr_length, const uint64_t rec_length) { + if (DEBUG_MODE) { + cout << "Decompressed data - " << type << endl; + cout << "Header length: " << hdr_length << endl; + if (rdres.zlib_perfect) { + cout << "ZLIB Parameters: compression level " << rdres.zlib_comp_level + << " memory level " << rdres.zlib_mem_level + << " window bits " << rdres.zlib_window_bits << endl; + } else { + cout << "Reconstruction data size: " << rdres.recon_data.size() << endl; + } + if (rec_length > 0) { + cout << "Recursion data length: " << rec_length << endl; + } else { + cout << "Recompressed length: " << rdres.compressed_stream_size << " - decompressed length: " << rdres.uncompressed_stream_size << endl; + } + } +} + +class OwnFileInputStream : public InputStream { +public: + OwnFileInputStream(FILE* f) : _f(f), _eof(false) {} + + virtual bool eof() const { + return _eof; + } + virtual size_t read(unsigned char* buffer, const size_t size) { + size_t res = own_fread(buffer, 1, size, _f); + _eof |= res < size; + return res; + } +private: + FILE* _f; + bool _eof; +}; +class UncompressedOutStream : public OutputStream { +public: + UncompressedOutStream(bool& in_memory) : _written(0), _in_memory(in_memory) {} + ~UncompressedOutStream() { + if (!_in_memory) { + safe_fclose(&ftempout); + } + } + + virtual size_t write(const unsigned char* buffer, const size_t size) { + print_work_sign(true); + if (_in_memory) { + if (_written + size >= MAX_IO_BUFFER_SIZE) { + _in_memory = false; + write_ftempout_if_not_present(_written, true, true); + } else { + memcpy(decomp_io_buf + _written, buffer, size); + _written += size; + return size; + } + } + return own_fwrite(buffer, 1, size, ftempout); + } +private: + size_t _written; + bool& _in_memory; +}; + +recompress_deflate_result try_recompression_deflate(FILE* file) { + if (file == fin) { + seek_64(file, input_file_pos); + } else { + seek_64(file, 0); + } + + recompress_deflate_result result; + memset(&result, 0, sizeof(result)); + + OwnFileInputStream is(file); + + std::vector unpacked_output; + uint64_t compressed_stream_size = 0; + result.accepted = preflate_decode(unpacked_output, result.recon_data, + compressed_stream_size, is, []() { print_work_sign(true); }); + result.compressed_stream_size = compressed_stream_size; + result.uncompressed_stream_size = unpacked_output.size(); + { + result.uncompressed_in_memory = true; + UncompressedOutStream uos(result.uncompressed_in_memory); + uos.write(unpacked_output.data(), unpacked_output.size()); + } + return std::move(result); +} + +class OwnFileOutputStream : public OutputStream { +public: + OwnFileOutputStream(FILE* f) : _f(f) {} + + virtual size_t write(const unsigned char* buffer, const size_t size) { + return own_fwrite(buffer, 1, size, _f); + } +private: + FILE* _f; +}; + +bool try_reconstructing_deflate(FILE* fin, FILE* fout, const recompress_deflate_result& rdres) { + std::vector unpacked_output; + unpacked_output.resize(rdres.uncompressed_stream_size); + if ((int64_t)own_fread(unpacked_output.data(), 1, rdres.uncompressed_stream_size, fin) != rdres.uncompressed_stream_size) { + return false; + } + OwnFileOutputStream os(fout); + bool result = preflate_reencode(os, rdres.recon_data, unpacked_output, []() { print_work_sign(true); }); + return result; +} +bool try_reconstructing_deflate_skip(FILE* fin, FILE* fout, const recompress_deflate_result& rdres, const size_t read_part, const size_t skip_part) { + std::vector unpacked_output; + unpacked_output.resize(rdres.uncompressed_stream_size); + frs_offset = 0; + frs_skip_len = skip_part; + frs_line_len = read_part; + if ((int64_t)fread_skip(unpacked_output.data(), 1, rdres.uncompressed_stream_size, fin) != rdres.uncompressed_stream_size) { + return false; + } + OwnFileOutputStream os(fout); + return preflate_reencode(os, rdres.recon_data, unpacked_output, []() { print_work_sign(true); }); +} +class OwnFileOutputStreamMultiPNG : public OutputStream { +public: + OwnFileOutputStreamMultiPNG(FILE* f, + const size_t idat_count, + const uint32_t* idat_crcs, + const uint32_t* idat_lengths) + : _f(f), _idat_count(idat_count), _idat_crcs(idat_crcs), _idat_lengths(idat_lengths) + , _idat_idx(0), _to_read(_idat_lengths[0]) + {} + + virtual size_t write(const unsigned char* buffer_, const size_t size_) { + if (_idat_idx >= _idat_count) { + return 0; + } + size_t written = 0; + size_t size = size_; + const unsigned char* buffer = buffer_; + while (size > _to_read) { + own_fwrite(buffer, 1, _to_read, _f); + written += _to_read; + size -= _to_read; + buffer += _to_read; + ++_idat_idx; + if (_idat_idx >= _idat_count) { + return written; + } + unsigned char crc_out[4] = {_idat_crcs[_idat_idx] >> 24, _idat_crcs[_idat_idx] >> 16, _idat_crcs[_idat_idx] >> 8, _idat_crcs[_idat_idx] >> 0}; + own_fwrite(crc_out, 1, 4, _f); + _to_read = _idat_lengths[_idat_idx]; + unsigned char len_out[4] = {_to_read >> 24, _to_read >> 16, _to_read >> 8, _to_read >> 0}; + own_fwrite(len_out, 1, 4, _f); + own_fwrite("IDAT", 1, 4, _f); + } + own_fwrite(buffer, 1, size, _f); + written += size; + _to_read -= size; + return written; + } +private: + FILE* _f; + const size_t _idat_count; + const uint32_t* _idat_crcs; + const uint32_t* _idat_lengths; + size_t _idat_idx, _to_read; +}; +bool try_reconstructing_deflate_multipng(FILE* fin, FILE* fout, const recompress_deflate_result& rdres, + const size_t idat_count, const uint32_t* idat_crcs, const uint32_t* idat_lengths) { + std::vector unpacked_output; + unpacked_output.resize(rdres.uncompressed_stream_size); + if ((int64_t)own_fread(unpacked_output.data(), 1, rdres.uncompressed_stream_size, fin) != rdres.uncompressed_stream_size) { + return false; + } + OwnFileOutputStreamMultiPNG os(fout, idat_count, idat_crcs, idat_lengths); + return preflate_reencode(os, rdres.recon_data, unpacked_output, []() { print_work_sign(true); }); +} + +static uint64_t sum_compressed = 0, sum_uncompressed = 0, sum_recon = 0, sum_expansion = 0; +void debug_sums(const recompress_deflate_result& rdres) { + if (DEBUG_MODE) { + sum_compressed += rdres.compressed_stream_size; + sum_uncompressed += rdres.uncompressed_stream_size; + sum_expansion += rdres.uncompressed_stream_size - rdres.compressed_stream_size; + sum_recon += rdres.recon_data.size(); + printf("deflate sums: c %I64d, u %I64d, x %I64d, r %I64d, i %I64d, o %I64d\n", + sum_compressed, sum_uncompressed, sum_expansion, sum_recon, _ftelli64(fin), _ftelli64(fout)); + } +} +void debug_pos() { + if (DEBUG_MODE) { + printf("deflate pos: i %I64d, o %I64d\n", _ftelli64(fin), _ftelli64(fout)); + } +} void try_decompression_pdf(int windowbits, int pdf_header_length, int img_width, int img_height, int img_bpc) { init_decompression_variables(); int bmp_header_type = 0; // 0 = none, 1 = 8-bit, 2 = 24-bit - // try to decompress at current position - long long compressed_stream_size = -1; - bool in_memory; - retval = try_to_decompress(fin, windowbits, compressed_stream_size, in_memory); + // try to decompress at current position + recompress_deflate_result rdres = try_recompression_deflate(fin); - if (retval > 0) { // seems to be a zLib-Stream + if (rdres.uncompressed_stream_size > 0) { // seems to be a zLib-Stream - decompressed_streams_count++; - if (img_bpc == 8) { - decompressed_pdf_count_8_bit++; - } else { - decompressed_pdf_count++; - } + decompressed_streams_count++; + if (img_bpc == 8) { + decompressed_pdf_count_8_bit++; + } else { + decompressed_pdf_count++; + } + + debug_deflate_detected(rdres, "in PDF"); + if (rdres.accepted) { + recompressed_streams_count++; + recompressed_pdf_count++; + + non_zlib_was_used = true; + debug_sums(rdres); + + if (img_bpc == 8) { + if (best_identical_bytes_decomp == (img_width * img_height)) { + bmp_header_type = 1; if (DEBUG_MODE) { - print_debug_percent(); - cout << "Possible zLib-Stream in PDF found at position " << saved_input_file_pos << ", windowbits = " << -windowbits << endl; - cout << "Compressed size: " << compressed_stream_size << endl; - cout << "Can be decompressed to " << retval << " bytes" << endl; + printf("Image size did match (8 bit)\n"); } - for (int index = MTF.First(); index>=0; index=MTF.Next()){ - if (comp_mem_level_count[index] == -1) continue; - int comp_level = (index % 9) + 1; - int mem_level = (index / 9) + 1; + recompressed_pdf_count_8_bit++; + recompressed_pdf_count--; + } else if (best_identical_bytes_decomp == (img_width * img_height * 3)) { + bmp_header_type = 2; + if (DEBUG_MODE) { + printf("Image size did match (24 bit)\n"); + } + decompressed_pdf_count_8_bit--; + decompressed_pdf_count_24_bit++; + recompressed_pdf_count_24_bit++; + recompressed_pdf_count--; + } else { + if (DEBUG_MODE) { + printf("Image size didn't match with stream size\n"); + } + recompressed_pdf_count--; + } + } - try_recompress(fin, comp_level, mem_level, windowbits, compressed_stream_size, retval, in_memory); + // end uncompressed data - if (final_compression_found){ MTF.Update(); break; } + compressed_data_found = true; + end_uncompressed_data(); + + debug_pos(); + + // write compressed data header (PDF) without 12 first bytes + // (/FlateDecode) + + unsigned char bmp_c = 0; + + if (bmp_header_type == 1) { + // 8 Bit, Bit 7,6 = 01 + bmp_c = 64; + } else if (bmp_header_type == 2) { + // 24 Bit, Bit 7,6 = 10 + bmp_c = 128; + } + + fout_fput_deflate_hdr(D_PDF, bmp_c, rdres, in_buf + cb + 12, pdf_header_length - 12, false); + fout_fput_recon_data(rdres); + + // eventually write BMP header + + if (bmp_header_type > 0) { + + int i; + + fout_fputc('B'); + fout_fputc('M'); + // BMP size in bytes + int bmp_size = ((img_width+3) & -4) * img_height; + if (bmp_header_type == 2) bmp_size *= 3; + if (bmp_header_type == 1) { + bmp_size += 54 + 1024; + } else { + bmp_size += 54; + } + fout_fput32_little_endian(bmp_size); + + for (i = 0; i < 4; i++) { + fout_fputc(0); + } + fout_fputc(54); + if (bmp_header_type == 1) { + fout_fputc(4); + } else { + fout_fputc(0); + } + fout_fputc(0); + fout_fputc(0); + fout_fputc(40); + fout_fputc(0); + fout_fputc(0); + fout_fputc(0); + + fout_fput32_little_endian(img_width); + fout_fput32_little_endian(img_height); + + fout_fputc(1); + fout_fputc(0); + + if (bmp_header_type == 1) { + fout_fputc(8); + } else { + fout_fputc(24); + } + fout_fputc(0); + + for (i = 0; i < 4; i++) { + fout_fputc(0); + } + + if (bmp_header_type == 2) img_width *= 3; + + int datasize = ((img_width+3) & -4) * img_height; + if (bmp_header_type == 2) datasize *= 3; + fout_fput32_little_endian(datasize); + + for (i = 0; i < 16; i++) { + fout_fputc(0); + } + + if (bmp_header_type == 1) { + // write BMP palette + for (i = 0; i < 1024; i++) { + fout_fputc(0); + } + } + } + + // write decompressed data + + if ((bmp_header_type == 0) || ((img_width % 4) == 0)) { + fout_fput_uncompressed(rdres); + } else { + if (!rdres.uncompressed_in_memory) { + ftempout = fopen(tempfile1,"rb"); + if (ftempout == NULL) { + error(ERR_TEMP_FILE_DISAPPEARED); } - if ((best_identical_bytes > min_ident_size) && (best_identical_bytes < best_identical_bytes_decomp)) { - recompressed_streams_count++; - recompressed_pdf_count++; + fseek(ftempout, 0, SEEK_SET); + } - if (DEBUG_MODE) { - cout << "Best match with level combination " << best_compression << best_mem_level << ": " << best_identical_bytes << " bytes, decompressed to " << best_identical_bytes_decomp << " bytes" << endl; - } - - if (img_bpc == 8) { - if (best_identical_bytes_decomp == (img_width * img_height)) { - bmp_header_type = 1; - if (DEBUG_MODE) { - printf("Image size did match (8 bit)\n"); - } - recompressed_pdf_count_8_bit++; - recompressed_pdf_count--; - } else if (best_identical_bytes_decomp == (img_width * img_height * 3)) { - bmp_header_type = 2; - if (DEBUG_MODE) { - printf("Image size did match (24 bit)\n"); - } - decompressed_pdf_count_8_bit--; - decompressed_pdf_count_24_bit++; - recompressed_pdf_count_24_bit++; - recompressed_pdf_count--; - } else { - if (DEBUG_MODE) { - printf("Image size didn't match with stream size\n"); - } - recompressed_pdf_count--; - } - } - - if (!(comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9] == -1)) { - if (fast_mode) { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - for (int i = 0; i < 81; i++) { - if (i != ((best_compression - 1) + (best_mem_level - 1) * 9)) { - comp_mem_level_count[i] = -1; - } - } - anything_was_used = true; - } else { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - anything_was_used = true; - } - } - - // end uncompressed data - - compressed_data_found = true; - end_uncompressed_data(); - - // write compressed data header (PDF) without 12 first bytes - // (/FlateDecode) - - unsigned char bmp_c = 0; - - if (bmp_header_type == 1) { - // 8 Bit, Bit 7,6 = 01 - bmp_c = 64; - } else if (bmp_header_type == 2) { - // 24 Bit, Bit 7,6 = 10 - bmp_c = 128; - } - - if (best_penalty_bytes_len == 0) { - fout_fputc(1 + (best_compression << 2) + bmp_c); - } else { - fout_fputc(1 + 2 + (best_compression << 2) + bmp_c); - } - fout_fputc(0); // PDF - fout_fputc((((-windowbits) - 8) << 4) + best_mem_level); - - pdf_header_length -= 12; - - fout_fput24(pdf_header_length); - - own_fwrite(in_buf + cb + 12, 1, pdf_header_length, fout); - - // store penalty bytes, if any - if (best_penalty_bytes_len != 0) { - if (DEBUG_MODE) { - printf("Penalty bytes were used: %i bytes\n", best_penalty_bytes_len); - } - fout_fput32(best_penalty_bytes_len); - for (int pbc = 0; pbc < best_penalty_bytes_len; pbc++) { - fout_fputc(best_penalty_bytes[pbc]); - } - } - - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); - - // eventually write BMP header - - if (bmp_header_type > 0) { - - int i; - - fout_fputc('B'); - fout_fputc('M'); - // BMP size in bytes - int bmp_size = ((img_width+3) & -4) * img_height; - if (bmp_header_type == 2) bmp_size *= 3; - if (bmp_header_type == 1) { - bmp_size += 54 + 1024; - } else { - bmp_size += 54; - } - fout_fput32_little_endian(bmp_size); - - for (i = 0; i < 4; i++) { - fout_fputc(0); - } - fout_fputc(54); - if (bmp_header_type == 1) { - fout_fputc(4); - } else { - fout_fputc(0); - } - fout_fputc(0); - fout_fputc(0); - fout_fputc(40); - fout_fputc(0); - fout_fputc(0); - fout_fputc(0); - - fout_fput32_little_endian(img_width); - fout_fput32_little_endian(img_height); - - fout_fputc(1); - fout_fputc(0); - - if (bmp_header_type == 1) { - fout_fputc(8); - } else { - fout_fputc(24); - } - fout_fputc(0); - - for (i = 0; i < 4; i++) { - fout_fputc(0); - } - - if (bmp_header_type == 2) img_width *= 3; - - int datasize = ((img_width+3) & -4) * img_height; - if (bmp_header_type == 2) datasize *= 3; - fout_fput32_little_endian(datasize); - - for (i = 0; i < 16; i++) { - fout_fputc(0); - } - - if (bmp_header_type == 1) { - // write BMP palette - for (i = 0; i < 1024; i++) { - fout_fputc(0); - } - } - } - - // write decompressed data - - if ((bmp_header_type == 0) || ((img_width % 4) == 0)) { - write_decompressed_data_io_buf(best_identical_bytes_decomp, in_memory); - } else { - if (!in_memory) { - ftempout = fopen(tempfile1,"rb"); - if (ftempout == NULL) { - error(ERR_TEMP_FILE_DISAPPEARED); - } - - fseek(ftempout, 0, SEEK_SET); - } - - unsigned char* buf_ptr = decomp_io_buf; - for (int y = 0; y < img_height; y++) { - - if (in_memory) { - fast_copy(buf_ptr, fout, img_width); - buf_ptr += img_width; - } else { - fast_copy(ftempout, fout, img_width); - } - - for (int i = 0; i < (4 - (img_width % 4)); i++) { - fout_fputc(0); - } - - } - - safe_fclose(&ftempout); - } - - // start new uncompressed data - - // set input file pointer after recompressed data - input_file_pos += best_identical_bytes - 1; - cb += best_identical_bytes - 1; + unsigned char* buf_ptr = decomp_io_buf; + for (int y = 0; y < img_height; y++) { + if (rdres.uncompressed_in_memory) { + fast_copy(buf_ptr, fout, img_width); + buf_ptr += img_width; } else { - if (intense_mode_is_active()) intense_ignore_offsets->insert(input_file_pos - 2); - if (brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); - if (DEBUG_MODE) { - printf("No matches\n"); - } + fast_copy(ftempout, fout, img_width); + } + + for (int i = 0; i < (4 - (img_width % 4)); i++) { + fout_fputc(0); } } + safe_fclose(&ftempout); + } + // start new uncompressed data + debug_pos(); + + // set input file pointer after recompressed data + input_file_pos += rdres.compressed_stream_size - 1; + cb += rdres.compressed_stream_size - 1; + + } else { + if (intense_mode_is_active()) intense_ignore_offsets->insert(input_file_pos - 2); + if (brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); + if (DEBUG_MODE) { + printf("No matches\n"); + } + } + } +} + +void try_decompression_deflate_type(unsigned& dcounter, unsigned& rcounter, + const unsigned char type, + const unsigned char* hdr, const int hdr_length, const bool inc_last, + const char* debugname) { + init_decompression_variables(); + + // try to decompress at current position + recompress_deflate_result rdres = try_recompression_deflate(fin); + + if (rdres.uncompressed_stream_size > 0) { // seems to be a zLib-Stream + decompressed_streams_count++; + dcounter++; + + debug_deflate_detected(rdres, debugname); + + if (rdres.accepted) { + recompressed_streams_count++; + rcounter++; + + non_zlib_was_used = true; + + debug_sums(rdres); + + // end uncompressed data + + compressed_data_found = true; + end_uncompressed_data(); + + debug_pos(); + + // check recursion + recursion_result r = recursion_write_file_and_compress(rdres); + + // write compressed data header without first bytes + fout_fput_deflate_rec(type, rdres, hdr, hdr_length, inc_last, r); + + debug_pos(); + + // set input file pointer after recompressed data + input_file_pos += rdres.compressed_stream_size - 1; + cb += rdres.compressed_stream_size - 1; + + } else { + if (type == D_SWF && intense_mode_is_active()) intense_ignore_offsets->insert(input_file_pos - 2); + if (type != D_BRUTE && brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); + if (DEBUG_MODE) { + printf("No matches\n"); + } + } + + } } void try_decompression_zip(int zip_header_length) { - init_decompression_variables(); - - int windowbits; - - // try to decompress at current position - long long compressed_stream_size = -1; - bool in_memory; - retval = try_to_decompress(fin, -15, compressed_stream_size, in_memory); - - if (retval > 0) { // seems to be a zLib-Stream - - decompressed_streams_count++; - decompressed_zip_count++; - - if (DEBUG_MODE) { - print_debug_percent(); - cout << "Possible zLib-Stream in ZIP found at position " << saved_input_file_pos << endl; - cout << "Compressed size: " << compressed_stream_size << endl; - cout << "Can be decompressed to " << retval << " bytes" << endl; - } - - for (windowbits = -15; windowbits < -7; windowbits++) { - for (int index = MTF.First(); index>=0; index=MTF.Next()){ - if (comp_mem_level_count[index] == -1) continue; - int comp_level = (index % 9) + 1; - int mem_level = (index / 9) + 1; - - try_recompress(fin, comp_level, mem_level, windowbits, compressed_stream_size, retval, in_memory); - - if (final_compression_found){ MTF.Update(); break; } - } - if (final_compression_found) break; - } - - if ((best_identical_bytes > min_ident_size) && (best_identical_bytes < best_identical_bytes_decomp)) { - recompressed_streams_count++; - recompressed_zip_count++; - - windowbits = best_windowbits; - if (DEBUG_MODE) { - cout << "Best match with level combination " << best_compression << best_mem_level << ", windowbits = " << -windowbits << ": " << best_identical_bytes << " bytes, decompressed to " << best_identical_bytes_decomp << " bytes" << endl; - } - - if (!(comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9] == -1)) { - if (fast_mode) { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - for (int i = 0; i < 81; i++) { - if (i != ((best_compression - 1) + (best_mem_level - 1) * 9)) { - comp_mem_level_count[i] = -1; - } - } - anything_was_used = true; - } else { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - anything_was_used = true; - } - } - - // end uncompressed data - - compressed_data_found = true; - end_uncompressed_data(); - - // check recursion - write_ftempout_if_not_present(best_identical_bytes_decomp, in_memory); - recursion_result r = recursion_compress(best_identical_bytes, best_identical_bytes_decomp); - - // write compressed data header (ZIP) without 4 first bytes (PK..) - - int header_byte = 1 + (best_compression << 2); - if (best_penalty_bytes_len != 0) { - header_byte += 2; - } - if (r.success) { - header_byte += 128; - } - fout_fputc(header_byte); - fout_fputc(1); // ZIP - fout_fputc((((-windowbits) - 8) << 4) + best_mem_level); - - zip_header_length -= 4; - - fout_fput24(zip_header_length); - - own_fwrite(in_buf + cb + 4, 1, zip_header_length, fout); - - // store penalty bytes, if any - if (best_penalty_bytes_len != 0) { - if (DEBUG_MODE) { - printf("Penalty bytes were used: %i bytes\n", best_penalty_bytes_len); - } - fout_fput32(best_penalty_bytes_len); - for (int pbc = 0; pbc < best_penalty_bytes_len; pbc++) { - fout_fputc(best_penalty_bytes[pbc]); - } - } - - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); - - if (r.success) { - fout_fput64(r.file_length); - } - - // write decompressed data - if (r.success) { - write_decompressed_data(r.file_length, r.file_name); - remove(r.file_name); - delete[] r.file_name; - } else { - write_decompressed_data_io_buf(best_identical_bytes_decomp, in_memory); - } - - // set input file pointer after recompressed data - input_file_pos += best_identical_bytes - 1; - cb += best_identical_bytes - 1; - - } else { - if (brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); - if (DEBUG_MODE) { - printf("No matches\n"); - } - } - - } + try_decompression_deflate_type(decompressed_zip_count, recompressed_zip_count, + D_ZIP, in_buf + cb + 4, zip_header_length - 4, false, + "in ZIP"); } void show_used_levels() { @@ -3900,7 +3647,9 @@ bool compress_file(float min_percent, float max_percent) { printf("extra field length: %i\n", extra_field_length); } - if ((filename_length + extra_field_length) <= CHECKBUF_SIZE) { + if ((filename_length + extra_field_length) <= CHECKBUF_SIZE + && compressed_size > 0 && compressed_size != uncompressed_size + && in_buf[cb + 8] == 8 && in_buf[cb + 9] == 0) { int header_length = 30 + filename_length + extra_field_length; @@ -4240,7 +3989,7 @@ bool compress_file(float min_percent, float max_percent) { idat_lengths[0] += 2; input_file_pos += 6; - try_decompression_png_multi(-windowbits); + try_decompression_png_multi(fpng, -windowbits); cb += 6; safe_fclose(&fpng); @@ -4724,7 +4473,7 @@ while (fin_pos < fin_length) { unsigned char header1 = fin_fgetc(); if (header1 == 0) { // uncompressed data long long uncompressed_data_length; - uncompressed_data_length = fin_fget64(); + uncompressed_data_length = fin_fget_vlint(); if (uncompressed_data_length == 0) break; // end of PCF file, used by bZip2 compress-on-the-fly @@ -4738,59 +4487,23 @@ while (fin_pos < fin_length) { unsigned char headertype = fin_fgetc(); - if (headertype == 0) { // PDF recompression - if (DEBUG_MODE) { - printf("Decompressed data - PDF\n"); - } - - unsigned char header2 = fin_fgetc(); - - bool penalty_bytes_stored = ((header1 & 2) == 2); - int compression_level = (header1 >> 2) & 15; - int bmp_c = (header1 >> 6); - int windowbits = -(((header2 >> 4) & 15) + 8); - int memlevel = header2 & 15; - - if (DEBUG_MODE) { - printf("Compression level: %i\n", compression_level); - printf("Window size: %i\n", -windowbits); - printf("Memory level: %i\n", memlevel); - if (bmp_c == 1) printf("Skipping BMP header (8-Bit)\n"); - if (bmp_c == 2) printf("Skipping BMP header (24-Bit)\n"); - } - - int pdf_header_length; - pdf_header_length = (fin_fgetc() << 16); - pdf_header_length += (fin_fgetc() << 8); - pdf_header_length += fin_fgetc(); - + switch (headertype) { + case D_PDF: { // PDF recompression + recompress_deflate_result rdres; + unsigned hdr_length; // restore PDF header fprintf(fout, "/FlateDecode"); + fin_fget_deflate_hdr(rdres, header1, in, hdr_length, false); + fin_fget_recon_data(rdres); + int bmp_c = (header1 >> 6); - own_fread(in, 1, pdf_header_length, fin); - own_fwrite(in, 1, pdf_header_length, fout); - - // read penalty bytes - if (penalty_bytes_stored) { - penalty_bytes_len = (fin_fgetc() << 24); - penalty_bytes_len += (fin_fgetc() << 16); - penalty_bytes_len += (fin_fgetc() << 8); - penalty_bytes_len += fin_fgetc(); - own_fread(penalty_bytes, 1, penalty_bytes_len, fin); - } - - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); - + debug_deflate_reconstruct(rdres, "PDF", hdr_length, 0); if (DEBUG_MODE) { - cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; + if (bmp_c == 1) printf("Skipping BMP header (8-Bit)\n"); + if (bmp_c == 2) printf("Skipping BMP header (24-Bit)\n"); } - // keep file position for penalty bytes - long long old_fout_pos = tell_64(fout); - // read BMP header - int bmp_width = 0; switch (bmp_c) { @@ -4806,457 +4519,117 @@ while (fin_pos < fin_length) { if (bmp_c == 2) bmp_width *= 3; } + uint64_t read_part, skip_part; if ((bmp_c == 0) || ((bmp_width % 4) == 0)) { // recompress directly to fout - retval = def_part(fin, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - if (retval != Z_OK) { - printf("Error recompressing data!"); - cout << "retval = " << retval << endl; - exit(0); - } + read_part = rdres.uncompressed_stream_size; + skip_part = 0; } else { // lines aligned to 4 byte, skip those bytes // recompress directly to fout, but skipping bytes - - retval = def_part_skip(fin, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length, bmp_width); - - if (retval != Z_OK) { - printf("Error recompressing data!"); - cout << "retval = " << retval << endl; - exit(0); - } - + read_part = bmp_width; + skip_part = (-bmp_width) & 3; } - - if (penalty_bytes_stored) { - fflush(fout); - - long long fsave_fout_pos = tell_64(fout); - - int pb_pos = 0; - for (int pbc = 0; pbc < penalty_bytes_len; pbc += 5) { - pb_pos = ((unsigned char)penalty_bytes[pbc]) << 24; - pb_pos += ((unsigned char)penalty_bytes[pbc + 1]) << 16; - pb_pos += ((unsigned char)penalty_bytes[pbc + 2]) << 8; - pb_pos += (unsigned char)penalty_bytes[pbc + 3]; - - seek_64(fout, old_fout_pos + pb_pos); - own_fwrite(penalty_bytes + pbc + 4, 1, 1, fout); - } - - seek_64(fout, fsave_fout_pos); + if (!try_reconstructing_deflate_skip(fin, fout, rdres, read_part, skip_part)) { + printf("Error recompressing data!"); + exit(0); } - - } else if (headertype == 1) { // ZIP recompression - - if (DEBUG_MODE) { - printf("Decompressed data - ZIP\n"); - } - - unsigned char header2 = fin_fgetc(); - - bool penalty_bytes_stored = ((header1 & 2) == 2); - bool recursion_used = ((header1 & 128) == 128); - int compression_level = (header1 >> 2) & 15; - int windowbits = -(((header2 >> 4) & 15) + 8); - int memlevel = header2 & 15; - - if (DEBUG_MODE) { - printf("Compression level: %i\n", compression_level); - printf("Window size: %i\n", -windowbits); - printf("Memory level: %i\n", memlevel); - } - - int zip_header_length; - zip_header_length = (fin_fgetc() << 16); - zip_header_length += (fin_fgetc() << 8); - zip_header_length += fin_fgetc(); - - if (DEBUG_MODE) { - printf("ZIP header length: %i\n", zip_header_length); - } - + break; + } + case D_ZIP: { // ZIP recompression + recompress_deflate_result rdres; + unsigned hdr_length; + int64_t recursion_data_length; fputc('P', fout); fputc('K', fout); fputc(3, fout); fputc(4, fout); + bool ok = fin_fget_deflate_rec(rdres, header1, in, hdr_length, false, recursion_data_length); - own_fread(in, 1, zip_header_length, fin); - own_fwrite(in, 1, zip_header_length, fout); + debug_deflate_reconstruct(rdres, "ZIP", hdr_length, recursion_data_length); - // read penalty bytes - if (penalty_bytes_stored) { - penalty_bytes_len = (fin_fgetc() << 24); - penalty_bytes_len += (fin_fgetc() << 16); - penalty_bytes_len += (fin_fgetc() << 8); - penalty_bytes_len += fin_fgetc(); - own_fread(penalty_bytes, 1, penalty_bytes_len, fin); - } - - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); - - long long recursion_data_length = 0; - if (recursion_used) { - recursion_data_length = fin_fget64(); - } - - if (DEBUG_MODE) { - if (recursion_used) { - cout << "Recursion data length: " << recursion_data_length << endl; - } else { - cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; - } - } - - long long old_fout_pos = tell_64(fout); - - if (recursion_used) { - recursion_result r = recursion_decompress(recursion_data_length); - retval = def_part(r.frecurse, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - safe_fclose(&r.frecurse); - remove(r.file_name); - delete[] r.file_name; - } else { - retval = def_part(fin, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - } - - if (retval != Z_OK) { + if (!ok) { printf("Error recompressing data!"); - cout << "retval = " << retval << endl; exit(0); } - - if (penalty_bytes_stored) { - fflush(fout); - - long long fsave_fout_pos = tell_64(fout); - int pb_pos = 0; - for (int pbc = 0; pbc < penalty_bytes_len; pbc += 5) { - pb_pos = ((unsigned char)penalty_bytes[pbc]) << 24; - pb_pos += ((unsigned char)penalty_bytes[pbc + 1]) << 16; - pb_pos += ((unsigned char)penalty_bytes[pbc + 2]) << 8; - pb_pos += (unsigned char)penalty_bytes[pbc + 3]; - - seek_64(fout, old_fout_pos + pb_pos); - own_fwrite(penalty_bytes + pbc + 4, 1, 1, fout); - - } - - seek_64(fout, fsave_fout_pos); - } - - } else if (headertype == 2) { // GZip recompression - - if (DEBUG_MODE) { - printf("Decompressed data - GZip\n"); - } - - unsigned char header2 = fin_fgetc(); - - bool penalty_bytes_stored = ((header1 & 2) == 2); - bool recursion_used = ((header1 & 128) == 128); - int compression_level = (header1 >> 2) & 15; - int windowbits = -(((header2 >> 4) & 15) + 8); - int memlevel = header2 & 15; - - if (DEBUG_MODE) { - printf("Compression level: %i\n", compression_level); - printf("Window size: %i\n", -windowbits); - printf("Memory level: %i\n", memlevel); - } - - int gzip_header_length; - gzip_header_length = (fin_fgetc() << 16); - gzip_header_length += (fin_fgetc() << 8); - gzip_header_length += fin_fgetc(); - - if (DEBUG_MODE) { - printf("GZip header length: %i\n", gzip_header_length); - } - + break; + } + case D_GZIP: { // GZip recompression + recompress_deflate_result rdres; + unsigned hdr_length; + int64_t recursion_data_length; fputc(31, fout); fputc(139, fout); + bool ok = fin_fget_deflate_rec(rdres, header1, in, hdr_length, false, recursion_data_length); - own_fread(in, 1, gzip_header_length, fin); - own_fwrite(in, 1, gzip_header_length, fout); + debug_deflate_reconstruct(rdres, "GZIP", hdr_length, recursion_data_length); - // read penalty bytes - if (penalty_bytes_stored) { - penalty_bytes_len = (fin_fgetc() << 24); - penalty_bytes_len += (fin_fgetc() << 16); - penalty_bytes_len += (fin_fgetc() << 8); - penalty_bytes_len += fin_fgetc(); - own_fread(penalty_bytes, 1, penalty_bytes_len, fin); - } - - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); - - long long recursion_data_length = 0; - if (recursion_used) { - recursion_data_length = fin_fget64(); - } - - if (DEBUG_MODE) { - if (recursion_used) { - cout << "Recursion data length: " << recursion_data_length << endl; - } else { - cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; - } - } - - long long old_fout_pos = tell_64(fout); - - if (recursion_used) { - recursion_result r = recursion_decompress(recursion_data_length); - retval = def_part(r.frecurse, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - safe_fclose(&r.frecurse); - remove(r.file_name); - delete[] r.file_name; - } else { - retval = def_part(fin, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - } - - if (retval != Z_OK) { + if (!ok) { printf("Error recompressing data!"); - cout << "retval = " << retval << endl; exit(0); } - - if (penalty_bytes_stored) { - fflush(fout); - - long long fsave_fout_pos = tell_64(fout); - int pb_pos = 0; - for (int pbc = 0; pbc < penalty_bytes_len; pbc += 5) { - pb_pos = ((unsigned char)penalty_bytes[pbc]) << 24; - pb_pos += ((unsigned char)penalty_bytes[pbc + 1]) << 16; - pb_pos += ((unsigned char)penalty_bytes[pbc + 2]) << 8; - pb_pos += (unsigned char)penalty_bytes[pbc + 3]; - - seek_64(fout, old_fout_pos + pb_pos); - own_fwrite(penalty_bytes + pbc + 4, 1, 1, fout); - } - - seek_64(fout, fsave_fout_pos); - } - - } else if (headertype == 3) { // PNG recompression - - if (DEBUG_MODE) { - printf("Decompressed data - PNG\n"); - } - - unsigned char header2 = fin_fgetc(); - - bool penalty_bytes_stored = ((header1 & 2) == 2); - int compression_level = (header1 >> 2) & 15; - int windowbits = -(((header2 >> 4) & 15) + 8); - int memlevel = header2 & 15; - - if (DEBUG_MODE) { - printf("Compression level: %i\n", compression_level); - printf("Window size: %i\n", -windowbits); - printf("Memory level: %i\n", memlevel); - } - + break; + } + case D_PNG: { // PNG recompression + recompress_deflate_result rdres; + unsigned hdr_length; // restore IDAT fprintf(fout, "IDAT"); - // restore zLib header (decrease by 1) - own_fread(in, 1, 2, fin); - unsigned char decchar = *(in + 1) - 1; - own_fwrite(in, 1, 1, fout); - fputc(decchar, fout); + fin_fget_deflate_hdr(rdres, header1, in, hdr_length, true); + fin_fget_recon_data(rdres); + debug_sums(rdres); + debug_pos(); - // read penalty bytes - if (penalty_bytes_stored) { - penalty_bytes_len = (fin_fgetc() << 24); - penalty_bytes_len += (fin_fgetc() << 16); - penalty_bytes_len += (fin_fgetc() << 8); - penalty_bytes_len += fin_fgetc(); - own_fread(penalty_bytes, 1, penalty_bytes_len, fin); - } + debug_deflate_reconstruct(rdres, "PNG", hdr_length, 0); - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); - - if (DEBUG_MODE) { - cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; - } - - long long old_fout_pos = tell_64(fout); - - retval = def_part(fin, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - - if (retval != Z_OK) { + if (!try_reconstructing_deflate(fin, fout, rdres)) { printf("Error recompressing data!"); - cout << "retval = " << retval << endl; exit(0); } - - if (penalty_bytes_stored) { - fflush(fout); - - long long fsave_fout_pos = tell_64(fout); - - int pb_pos = 0; - for (int pbc = 0; pbc < penalty_bytes_len; pbc += 5) { - pb_pos = ((unsigned char)penalty_bytes[pbc]) << 24; - pb_pos += ((unsigned char)penalty_bytes[pbc + 1]) << 16; - pb_pos += ((unsigned char)penalty_bytes[pbc + 2]) << 8; - pb_pos += (unsigned char)penalty_bytes[pbc + 3]; - - seek_64(fout, old_fout_pos + pb_pos); - own_fwrite(penalty_bytes + pbc + 4, 1, 1, fout); - } - - seek_64(fout, fsave_fout_pos); - } - - } else if (headertype == 4) { // PNG multi recompression - - if (DEBUG_MODE) { - printf("Decompressed data - PNG multi\n"); - } - - unsigned char header2 = fin_fgetc(); - - bool penalty_bytes_stored = ((header1 & 2) == 2); - int compression_level = (header1 >> 2) & 15; - int windowbits = -(((header2 >> 4) & 15) + 8); - int memlevel = header2 & 15; - - if (DEBUG_MODE) { - printf("Compression level: %i\n", compression_level); - printf("Window size: %i\n", -windowbits); - printf("Memory level: %i\n", memlevel); - } - + debug_pos(); + break; + } + case D_MULTIPNG: { // PNG multi recompression + recompress_deflate_result rdres; + unsigned hdr_length; // restore first IDAT fprintf(fout, "IDAT"); - - // restore zLib header (decrease by 1) - own_fread(in, 1, 2, fin); - unsigned char decchar = *(in + 1) - 1; - own_fwrite(in, 1, 1, fout); - fputc(decchar, fout); + + fin_fget_deflate_hdr(rdres, header1, in, hdr_length, true); // get IDAT count - own_fread(in, 1, 2, fin); - idat_count = (in[0] << 8) + in[1]; - idat_count++; + idat_count = fin_fget_vlint() + 1; idat_crcs = (unsigned int*)(realloc(idat_crcs, idat_count * sizeof(unsigned int))); idat_lengths = (unsigned int*)(realloc(idat_lengths, idat_count * sizeof(unsigned int))); // get first IDAT length - own_fread(in, 1, 4, fin); - idat_lengths[0] = (in[0] << 24) + (in[1] << 16) + (in[2] << 8) + in[3]; - idat_lengths[0] -= 2; // zLib header length + idat_lengths[0] = fin_fget_vlint() - 2; // zLib header length // get IDAT chunk lengths and CRCs for (int i = 1; i < idat_count; i++) { - own_fread(in, 1, 4, fin); - idat_crcs[i] = (in[0] << 24) + (in[1] << 16) + (in[2] << 8) + in[3]; - own_fread(in, 1, 4, fin); - idat_lengths[i] = (in[0] << 24) + (in[1] << 16) + (in[2] << 8) + in[3]; + idat_crcs[i] = fin_fget32(); + idat_lengths[i] = fin_fget_vlint(); } - // read penalty bytes - if (penalty_bytes_stored) { - penalty_bytes_len = (fin_fgetc() << 24); - penalty_bytes_len += (fin_fgetc() << 16); - penalty_bytes_len += (fin_fgetc() << 8); - penalty_bytes_len += fin_fgetc(); - own_fread(penalty_bytes, 1, penalty_bytes_len, fin); - } + fin_fget_recon_data(rdres); + debug_sums(rdres); + debug_pos(); - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); + debug_deflate_reconstruct(rdres, "PNG multi", hdr_length, 0); - if (DEBUG_MODE) { - cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; - } - - remove(tempfile1); - ftempout = tryOpen(tempfile1,"wb"); - - fast_copy(fin, ftempout, decompressed_data_length); - - safe_fclose(&ftempout); - - remove(tempfile2); - - ftempout = tryOpen(tempfile1,"rb"); - frecomp = tryOpen(tempfile2,"wb"); - - long long old_frecomp_pos = tell_64(frecomp); - - // recompress data - retval = def(ftempout, frecomp, compression_level, windowbits, memlevel); - if ((!penalty_bytes_stored) || (retval != Z_OK)) safe_fclose(&frecomp); - safe_fclose(&ftempout); - - if (retval != Z_OK) { + if (!try_reconstructing_deflate_multipng(fin, fout, rdres, idat_count, idat_crcs, idat_lengths)) { printf("Error recompressing data!"); - cout << "retval = " << retval << endl; exit(0); } - - if (penalty_bytes_stored) { - fflush(frecomp); - - int pb_pos = 0; - for (int pbc = 0; pbc < penalty_bytes_len; pbc += 5) { - pb_pos = ((unsigned char)penalty_bytes[pbc]) << 24; - pb_pos += ((unsigned char)penalty_bytes[pbc + 1]) << 16; - pb_pos += ((unsigned char)penalty_bytes[pbc + 2]) << 8; - pb_pos += (unsigned char)penalty_bytes[pbc + 3]; - - seek_64(frecomp, old_frecomp_pos + pb_pos); - own_fwrite(penalty_bytes + pbc + 4, 1, 1, frecomp); - } - - safe_fclose(&frecomp); - } - - frecomp = tryOpen(tempfile2,"rb"); - - long long remaining_bytes = recompressed_data_length; - unsigned int act_idat_chunk = 0; - for (;;) { - if ((remaining_bytes + 2) > (int)idat_lengths[act_idat_chunk]) { - fast_copy(frecomp, fout, idat_lengths[act_idat_chunk]); - remaining_bytes -= idat_lengths[act_idat_chunk]; - - fputc((idat_crcs[act_idat_chunk+1] >> 24) % 256, fout); - fputc((idat_crcs[act_idat_chunk+1] >> 16) % 256, fout); - fputc((idat_crcs[act_idat_chunk+1] >> 8) % 256, fout); - fputc(idat_crcs[act_idat_chunk+1] % 256, fout); - fputc((idat_lengths[act_idat_chunk+1] >> 24) % 256, fout); - fputc((idat_lengths[act_idat_chunk+1] >> 16) % 256, fout); - fputc((idat_lengths[act_idat_chunk+1] >> 8) % 256, fout); - fputc(idat_lengths[act_idat_chunk+1] % 256, fout); - fprintf(fout, "IDAT"); - } else { - fast_copy(frecomp, fout, remaining_bytes); - break; - } - act_idat_chunk++; - } - - safe_fclose(&frecomp); - - remove(tempfile2); - remove(tempfile1); - + debug_pos(); free(idat_lengths); idat_lengths = NULL; free(idat_crcs); idat_crcs = NULL; - - } else if (headertype == 5) { // GIF recompression + break; + } + case D_GIF: { // GIF recompression if (DEBUG_MODE) { printf("Decompressed data - GIF\n"); @@ -5271,10 +4644,7 @@ while (fin_pos < fin_length) { GifDiffStruct gDiff; // read diff bytes - gDiff.GIFDiffIndex = (fin_fgetc() << 24); - gDiff.GIFDiffIndex += (fin_fgetc() << 16); - gDiff.GIFDiffIndex += (fin_fgetc() << 8); - gDiff.GIFDiffIndex += fin_fgetc(); + gDiff.GIFDiffIndex = fin_fget_vlint(); gDiff.GIFDiff = (unsigned char*)malloc(gDiff.GIFDiffIndex * sizeof(unsigned char)); own_fread(gDiff.GIFDiff, 1, gDiff.GIFDiffIndex, fin); if (DEBUG_MODE) { @@ -5286,15 +4656,12 @@ while (fin_pos < fin_length) { // read penalty bytes if (penalty_bytes_stored) { - penalty_bytes_len = (fin_fgetc() << 24); - penalty_bytes_len += (fin_fgetc() << 16); - penalty_bytes_len += (fin_fgetc() << 8); - penalty_bytes_len += fin_fgetc(); + penalty_bytes_len = fin_fget_vlint(); own_fread(penalty_bytes, 1, penalty_bytes_len, fin); } - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); + long long recompressed_data_length = fin_fget_vlint(); + long long decompressed_data_length = fin_fget_vlint(); if (DEBUG_MODE) { cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; @@ -5359,8 +4726,9 @@ while (fin_pos < fin_length) { } GifDiffFree(&gDiff); - - } else if (headertype == 6) { // JPG recompression + break; + } + case D_JPG: { // JPG recompression if (DEBUG_MODE) { printf("Decompressed data - JPG\n"); @@ -5368,8 +4736,8 @@ while (fin_pos < fin_length) { bool mjpg_dht_used = ((header1 & 4) == 4); - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); + long long recompressed_data_length = fin_fget_vlint(); + long long decompressed_data_length = fin_fget_vlint(); if (DEBUG_MODE) { cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; @@ -5478,107 +4846,26 @@ while (fin_pos < fin_length) { remove(tempfile2); remove(tempfile1); } - - } else if (headertype == 7) { // SWF recompression - - if (DEBUG_MODE) { - printf("Decompressed data - SWF\n"); - } - - unsigned char header2 = fin_fgetc(); - - bool penalty_bytes_stored = ((header1 & 2) == 2); - bool recursion_used = ((header1 & 128) == 128); - int compression_level = (header1 >> 2) & 15; - int windowbits = -(((header2 >> 4) & 15) + 8); - int memlevel = header2 & 15; - - if (DEBUG_MODE) { - printf("Compression level: %i\n", compression_level); - printf("Window size: %i\n", -windowbits); - printf("Memory level: %i\n", memlevel); - } - + break; + } + case D_SWF: { // SWF recompression + recompress_deflate_result rdres; + unsigned hdr_length; + int64_t recursion_data_length; fputc('C', fout); fputc('W', fout); fputc('S', fout); - // get Flash version - char c = fin_fgetc(); - fputc(c, fout); - // get length from SWF header - for (int i = 0; i < 4; i++) { - c = fin_fgetc(); - fputc(c, fout); - } + bool ok = fin_fget_deflate_rec(rdres, header1, in, hdr_length, true, recursion_data_length); - // restore zLib header (decrease by 1) - own_fread(in, 1, 2, fin); - unsigned char decchar = *(in + 1) - 1; - own_fwrite(in, 1, 1, fout); - fputc(decchar, fout); + debug_deflate_reconstruct(rdres, "SWF", hdr_length, recursion_data_length); - // read penalty bytes - if (penalty_bytes_stored) { - penalty_bytes_len = (fin_fgetc() << 24); - penalty_bytes_len += (fin_fgetc() << 16); - penalty_bytes_len += (fin_fgetc() << 8); - penalty_bytes_len += fin_fgetc(); - own_fread(penalty_bytes, 1, penalty_bytes_len, fin); - } - - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); - - long long recursion_data_length = 0; - if (recursion_used) { - recursion_data_length = fin_fget64(); - } - - if (DEBUG_MODE) { - if (recursion_used) { - cout << "Recursion data length: " << recursion_data_length << endl; - } else { - cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; - } - } - - long long old_fout_pos = tell_64(fout); - - if (recursion_used) { - recursion_result r = recursion_decompress(recursion_data_length); - retval = def_part(r.frecurse, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - safe_fclose(&r.frecurse); - remove(r.file_name); - delete[] r.file_name; - } else { - retval = def_part(fin, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - } - - if (retval != Z_OK) { + if (!ok) { printf("Error recompressing data!"); - cout << "retval = " << retval << endl; exit(0); } - - if (penalty_bytes_stored) { - fflush(fout); - - long long fsave_fout_pos = tell_64(fout); - int pb_pos = 0; - for (int pbc = 0; pbc < penalty_bytes_len; pbc += 5) { - pb_pos = ((unsigned char)penalty_bytes[pbc]) << 24; - pb_pos += ((unsigned char)penalty_bytes[pbc + 1]) << 16; - pb_pos += ((unsigned char)penalty_bytes[pbc + 2]) << 8; - pb_pos += (unsigned char)penalty_bytes[pbc + 3]; - - seek_64(fout, old_fout_pos + pb_pos); - own_fwrite(penalty_bytes + pbc + 4, 1, 1, fout); - } - - seek_64(fout, fsave_fout_pos); - } - - } else if (headertype == 8) { // Base64 recompression + break; + } + case D_BASE64: { // Base64 recompression if (DEBUG_MODE) { printf("Decompressed data - Base64\n"); @@ -5588,20 +4875,17 @@ while (fin_pos < fin_length) { bool recursion_used = ((header1 & 128) == 128); // restore Base64 "header" - int base64_header_length; - base64_header_length = (fin_fgetc() << 8); - base64_header_length += fin_fgetc(); + int base64_header_length = fin_fget_vlint(); if (DEBUG_MODE) { - printf("Base64 header length: %i\n", base64_header_length); + printf("Base64 header length: %i\n", base64_header_length); } own_fread(in, 1, base64_header_length, fin); fputc(*(in) + 1, fout); // first char was decreased own_fwrite(in + 1, 1, base64_header_length - 1, fout); // read line length list - int line_count = fin_fgetc() << 8; - line_count += fin_fgetc(); + int line_count = fin_fget_vlint(); unsigned int* base64_line_len = new unsigned int[line_count]; @@ -5617,12 +4901,12 @@ while (fin_pos < fin_length) { if (line_case == 1) base64_line_len[line_count - 1] = fin_fgetc(); } - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); + long long recompressed_data_length = fin_fget_vlint(); + long long decompressed_data_length = fin_fget_vlint(); long long recursion_data_length = 0; if (recursion_used) { - recursion_data_length = fin_fget64(); + recursion_data_length = fin_fget_vlint(); } if (DEBUG_MODE) { @@ -5646,7 +4930,9 @@ while (fin_pos < fin_length) { } delete[] base64_line_len; - } else if (headertype == 9) { // bZip2 recompression + break; + } + case D_BZIP2: { // bZip2 recompression if (DEBUG_MODE) { printf("Decompressed data - bZip2\n"); @@ -5664,19 +4950,16 @@ while (fin_pos < fin_length) { // read penalty bytes if (penalty_bytes_stored) { - penalty_bytes_len = (fin_fgetc() << 24); - penalty_bytes_len += (fin_fgetc() << 16); - penalty_bytes_len += (fin_fgetc() << 8); - penalty_bytes_len += fin_fgetc(); + penalty_bytes_len = fin_fget_vlint(); own_fread(penalty_bytes, 1, penalty_bytes_len, fin); } - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); + long long recompressed_data_length = fin_fget_vlint(); + long long decompressed_data_length = fin_fget_vlint(); long long recursion_data_length = 0; if (recursion_used) { - recursion_data_length = fin_fget64(); + recursion_data_length = fin_fget_vlint(); } if (DEBUG_MODE) { @@ -5722,14 +5005,16 @@ while (fin_pos < fin_length) { seek_64(fout, fsave_fout_pos); } - } else if (headertype == 10) { // MP3 recompression + break; + } + case D_MP3: { // MP3 recompression if (DEBUG_MODE) { printf("Decompressed data - MP3\n"); } - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); + long long recompressed_data_length = fin_fget_vlint(); + long long decompressed_data_length = fin_fget_vlint(); if (DEBUG_MODE) { cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; @@ -5784,175 +5069,37 @@ while (fin_pos < fin_length) { remove(tempfile2); remove(tempfile1); } - } else if (headertype == 254) { // brute mode recompression + break; + } + case D_BRUTE: { // brute mode recompression + recompress_deflate_result rdres; + unsigned hdr_length; + int64_t recursion_data_length; + bool ok = fin_fget_deflate_rec(rdres, header1, in, hdr_length, false, recursion_data_length); - if (DEBUG_MODE) { - printf("Decompressed data - brute mode\n"); - } + debug_deflate_reconstruct(rdres, "brute mode", hdr_length, recursion_data_length); - unsigned char header2 = fin_fgetc(); - - bool penalty_bytes_stored = ((header1 & 2) == 2); - bool recursion_used = ((header1 & 128) == 128); - int compression_level = (header1 >> 2) & 15; - int windowbits = -(((header2 >> 4) & 15) + 8); - int memlevel = header2 & 15; - - if (DEBUG_MODE) { - printf("Compression level: %i\n", compression_level); - printf("Window size: %i\n", -windowbits); - printf("Memory level: %i\n", memlevel); - } - - // read penalty bytes - if (penalty_bytes_stored) { - penalty_bytes_len = (fin_fgetc() << 24); - penalty_bytes_len += (fin_fgetc() << 16); - penalty_bytes_len += (fin_fgetc() << 8); - penalty_bytes_len += fin_fgetc(); - own_fread(penalty_bytes, 1, penalty_bytes_len, fin); - } - - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); - - long long recursion_data_length = 0; - if (recursion_used) { - recursion_data_length = fin_fget64(); - } - - if (DEBUG_MODE) { - if (recursion_used) { - cout << "Recursion data length: " << recursion_data_length << endl; - } else { - cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; - } - } - - long long old_fout_pos = tell_64(fout); - - if (recursion_used) { - recursion_result r = recursion_decompress(recursion_data_length); - retval = def_part(r.frecurse, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - safe_fclose(&r.frecurse); - remove(r.file_name); - delete[] r.file_name; - } else { - retval = def_part(fin, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - } - - if (retval != Z_OK) { + if (!ok) { printf("Error recompressing data!"); - cout << "retval = " << retval << endl; exit(0); } + break; + } + case D_RAW: { // raw zLib recompression + recompress_deflate_result rdres; + unsigned hdr_length; + int64_t recursion_data_length; + bool ok = fin_fget_deflate_rec(rdres, header1, in, hdr_length, true, recursion_data_length); - if (penalty_bytes_stored) { - fflush(fout); + debug_deflate_reconstruct(rdres, "raw zLib", hdr_length, recursion_data_length); - long long fsave_fout_pos = tell_64(fout); - int pb_pos = 0; - for (int pbc = 0; pbc < penalty_bytes_len; pbc += 5) { - pb_pos = ((unsigned char)penalty_bytes[pbc]) << 24; - pb_pos += ((unsigned char)penalty_bytes[pbc + 1]) << 16; - pb_pos += ((unsigned char)penalty_bytes[pbc + 2]) << 8; - pb_pos += (unsigned char)penalty_bytes[pbc + 3]; - - seek_64(fout, old_fout_pos + pb_pos); - own_fwrite(penalty_bytes + pbc + 4, 1, 1, fout); - } - - seek_64(fout, fsave_fout_pos); - } - - } else if (headertype == 255) { // raw zLib recompression - - if (DEBUG_MODE) { - printf("Decompressed data - raw zLib\n"); - } - - unsigned char header2 = fin_fgetc(); - - bool penalty_bytes_stored = ((header1 & 2) == 2); - bool recursion_used = ((header1 & 128) == 128); - int compression_level = (header1 >> 2) & 15; - int windowbits = -(((header2 >> 4) & 15) + 8); - int memlevel = header2 & 15; - - if (DEBUG_MODE) { - printf("Compression level: %i\n", compression_level); - printf("Window size: %i\n", -windowbits); - printf("Memory level: %i\n", memlevel); - } - - // restore zLib header (decrease by 1) - own_fread(in, 1, 2, fin); - unsigned char decchar = *(in + 1) - 1; - own_fwrite(in, 1, 1, fout); - fputc(decchar, fout); - - // read penalty bytes - if (penalty_bytes_stored) { - penalty_bytes_len = (fin_fgetc() << 24); - penalty_bytes_len += (fin_fgetc() << 16); - penalty_bytes_len += (fin_fgetc() << 8); - penalty_bytes_len += fin_fgetc(); - own_fread(penalty_bytes, 1, penalty_bytes_len, fin); - } - - long long recompressed_data_length = fin_fget64(); - long long decompressed_data_length = fin_fget64(); - - long long recursion_data_length = 0; - if (recursion_used) { - recursion_data_length = fin_fget64(); - } - - if (DEBUG_MODE) { - if (recursion_used) { - cout << "Recursion data length: " << recursion_data_length << endl; - } else { - cout << "Recompressed length: " << recompressed_data_length << " - decompressed length: " << decompressed_data_length << endl; - } - } - - long long old_fout_pos = tell_64(fout); - - if (recursion_used) { - recursion_result r = recursion_decompress(recursion_data_length); - retval = def_part(r.frecurse, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - safe_fclose(&r.frecurse); - remove(r.file_name); - delete[] r.file_name; - } else { - retval = def_part(fin, fout, compression_level, windowbits, memlevel, decompressed_data_length, recompressed_data_length); - } - - if (retval != Z_OK) { + if (!ok) { printf("Error recompressing data!"); - cout << "retval = " << retval << endl; exit(0); } - - if (penalty_bytes_stored) { - fflush(fout); - - long long fsave_fout_pos = tell_64(fout); - - int pb_pos = 0; - for (int pbc = 0; pbc < penalty_bytes_len; pbc += 5) { - pb_pos = ((unsigned char)penalty_bytes[pbc]) << 24; - pb_pos += ((unsigned char)penalty_bytes[pbc + 1]) << 16; - pb_pos += ((unsigned char)penalty_bytes[pbc + 2]) << 8; - pb_pos += (unsigned char)penalty_bytes[pbc + 3]; - - seek_64(fout, old_fout_pos + pb_pos); - own_fwrite(penalty_bytes + pbc + 4, 1, 1, fout); - } - - seek_64(fout, fsave_fout_pos); - } - } else { + break; + } + default: printf("ERROR: Unsupported stream type %i\n", headertype); exit(0); } @@ -6015,24 +5162,6 @@ void convert_file() { denit_convert(); } -long long try_to_decompress(FILE* file, int windowbits, long long& compressed_stream_size, bool& in_memory) { - long long r, decompressed_stream_size; - - print_work_sign(true); - - if (file == fin) { - seek_64(file, input_file_pos); - } else { - seek_64(file, 0); - } - - in_memory = true; - r = inf(file, windowbits, compressed_stream_size, decompressed_stream_size, in_memory); - if (r == Z_OK) return decompressed_stream_size; - - return r; -} - long long try_to_decompress_bzip2(FILE* file, int compression_level, long long& compressed_stream_size) { long long r, decompressed_stream_size; @@ -6053,58 +5182,6 @@ long long try_to_decompress_bzip2(FILE* file, int compression_level, long long& return r; } -void try_recompress(FILE* origfile, int comp_level, int mem_level, int windowbits, long long& compressed_stream_size, long long decomp_bytes_total, bool in_memory) { - print_work_sign(true); - - identical_bytes = file_recompress(origfile, comp_level, windowbits, mem_level, identical_bytes_decomp, decomp_bytes_total, in_memory); - if (identical_bytes > -1) { // successfully recompressed? - if ((identical_bytes > best_identical_bytes) || ((identical_bytes == best_identical_bytes) && (penalty_bytes_len < best_penalty_bytes_len))) { - if (identical_bytes > min_ident_size) { - if (DEBUG_MODE) { - cout << "Identical recompressed bytes: " << identical_bytes << " of " << compressed_stream_size << endl; - cout << "Identical decompressed bytes: " << identical_bytes_decomp << " of " << decomp_bytes_total << endl; - } - - bool enough_identical_compressed_bytes = (identical_bytes == compressed_stream_size); - if (!enough_identical_compressed_bytes) { - if ((identical_bytes > DEF_COMPARE_CHUNK) && (identical_bytes + IDENTICAL_COMPRESSED_BYTES_TOLERANCE >= compressed_stream_size)) { - enough_identical_compressed_bytes = true; - } - } - - final_compression_found = (identical_bytes_decomp == decomp_bytes_total) && (enough_identical_compressed_bytes) && (penalty_bytes_len < PENALTY_BYTES_TOLERANCE); - - // Partial matches sometimes need all the decompressed bytes, but there are much less - // identical recompressed bytes - in these cases, all the decompressed bytes have to - // be stored together with the remaining recompressed bytes, so the result won't compress - // better than the original stream. What's important here is the ratio between recompressed ratio - // and decompressed ratio that shouldn't get too high. - // Example: A stream has 5 of 1000 identical recompressed bytes, but needs 1000 of 1000 decompressed bytes, - // so the ratio is (1000/1000)/(5/1000) = 200 which is too high. With 5 of 1000 decompressed bytes or - // 1000 of 1000 identical recompressed bytes, ratio would've been 1 and we'd accept it. - float partial_ratio = ((float)identical_bytes_decomp / decomp_bytes_total) / ((float)identical_bytes / compressed_stream_size); - if (partial_ratio < 3.0f) { - best_identical_bytes_decomp = identical_bytes_decomp; - best_identical_bytes = identical_bytes; - best_compression = comp_level; - best_mem_level = mem_level; - best_windowbits = windowbits; - if (penalty_bytes_len > 0) { - memcpy(best_penalty_bytes, penalty_bytes, penalty_bytes_len); - best_penalty_bytes_len = penalty_bytes_len; - } else { - best_penalty_bytes_len = 0; - } - } else { - if (DEBUG_MODE) { - printf("Not enough identical recompressed bytes\n"); - } - } - } - } - } -} - void try_recompress_bzip2(FILE* origfile, int level, long long& compressed_stream_size) { print_work_sign(true); @@ -6697,404 +5774,149 @@ long long compare_files_penalty(FILE* file1, FILE* file2, long long pos1, long l } void try_decompression_gzip(int gzip_header_length) { - init_decompression_variables(); - - int windowbits; - - // try to decompress at current position - long long compressed_stream_size = -1; - bool in_memory; - retval = try_to_decompress(fin, -15, compressed_stream_size, in_memory); - - if (retval > 0) { // seems to be a zLib-Stream - - decompressed_streams_count++; - decompressed_gzip_count++; - - if (DEBUG_MODE) { - print_debug_percent(); - cout << "Possible zLib-Stream in GZip found at position " << saved_input_file_pos << endl; - cout << "Compressed size: " << compressed_stream_size << endl; - cout << "Can be decompressed to " << retval << " bytes" << endl; - } - - for (windowbits = -15; windowbits < -7; windowbits++) { - for (int index = MTF.First(); index>=0; index=MTF.Next()){ - if (comp_mem_level_count[index] == -1) continue; - int comp_level = (index % 9) + 1; - int mem_level = (index / 9) + 1; - - try_recompress(fin, comp_level, mem_level, windowbits, compressed_stream_size, retval, in_memory); - - if (final_compression_found){ MTF.Update(); break; } - } - if (final_compression_found) break; - } - - if ((best_identical_bytes > min_ident_size) && (best_identical_bytes < best_identical_bytes_decomp)) { - recompressed_streams_count++; - recompressed_gzip_count++; - - windowbits = best_windowbits; - if (DEBUG_MODE) { - cout << "Best match with level combination " << best_compression << best_mem_level << ", windowbits = " << -windowbits << ": " << best_identical_bytes << " bytes, decompressed to " << best_identical_bytes_decomp << " bytes" << endl; - } - - if (!(comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9] == -1)) { - if (fast_mode) { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - for (int i = 0; i < 81; i++) { - if (i != ((best_compression - 1) + (best_mem_level - 1) * 9)) { - comp_mem_level_count[i] = -1; - } - } - anything_was_used = true; - } else { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - anything_was_used = true; - } - } - - // end uncompressed data - - compressed_data_found = true; - end_uncompressed_data(); - - // check recursion - write_ftempout_if_not_present(best_identical_bytes_decomp, in_memory); - recursion_result r = recursion_compress(best_identical_bytes, best_identical_bytes_decomp); - - // write compressed data header (GZip) without 2 first bytes - - int header_byte = 1 + (best_compression << 2); - if (best_penalty_bytes_len != 0) { - header_byte += 2; - } - if (r.success) { - header_byte += 128; - } - fout_fputc(header_byte); - fout_fputc(2); // GZip - fout_fputc((((-windowbits) - 8) << 4) + best_mem_level); - - gzip_header_length -= 2; - - fout_fput24(gzip_header_length); - - own_fwrite(in_buf + cb + 2, 1, gzip_header_length, fout); - - // store penalty bytes, if any - if (best_penalty_bytes_len != 0) { - if (DEBUG_MODE) { - printf("Penalty bytes were used: %i bytes\n", best_penalty_bytes_len); - } - fout_fput32(best_penalty_bytes_len); - for (int pbc = 0; pbc < best_penalty_bytes_len; pbc++) { - fout_fputc(best_penalty_bytes[pbc]); - } - } - - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); - - if (r.success) { - fout_fput64(r.file_length); - } - - // write decompressed data - if (r.success) { - write_decompressed_data(r.file_length, r.file_name); - remove(r.file_name); - delete[] r.file_name; - } else { - write_decompressed_data_io_buf(best_identical_bytes_decomp, in_memory); - } - - // start new uncompressed data - - // set input file pointer after recompressed data - input_file_pos += best_identical_bytes - 1; - cb += best_identical_bytes - 1; - - } else { - if (brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); - if (DEBUG_MODE) { - printf("No matches\n"); - } - } - - } - - + try_decompression_deflate_type(decompressed_gzip_count, recompressed_gzip_count, + D_GZIP, in_buf + cb + 2, gzip_header_length - 2, false, + "in GZIP"); } void try_decompression_png (int windowbits) { init_decompression_variables(); - // try to decompress at current position - long long compressed_stream_size = -1; - bool in_memory; - retval = try_to_decompress(fin, windowbits, compressed_stream_size, in_memory); + // try to decompress at current position + recompress_deflate_result rdres = try_recompression_deflate(fin); - if (retval > 0) { // seems to be a zLib-Stream + if (rdres.uncompressed_stream_size > 0) { // seems to be a zLib-Stream - decompressed_streams_count++; - decompressed_png_count++; + decompressed_streams_count++; + decompressed_png_count++; - if (DEBUG_MODE) { - print_debug_percent(); - cout << "Possible zLib-Stream in PNG found at position " << saved_input_file_pos << ", windowbits = " << -windowbits << endl; - cout << "Compressed size: " << compressed_stream_size << endl; - cout << "Can be decompressed to " << retval << " bytes" << endl; - } + debug_deflate_detected(rdres, "in PNG"); - for (int index = MTF.First(); index>=0; index=MTF.Next()){ - if (comp_mem_level_count[index] == -1) continue; - int comp_level = (index % 9) + 1; - int mem_level = (index / 9) + 1; + if (rdres.accepted) { + recompressed_streams_count++; + recompressed_png_count++; - try_recompress(fin, comp_level, mem_level, windowbits, compressed_stream_size, retval, in_memory); + non_zlib_was_used = true; - if (final_compression_found){ MTF.Update(); break; } - } + debug_sums(rdres); - if ((best_identical_bytes > min_ident_size) && (best_identical_bytes < best_identical_bytes_decomp)) { - recompressed_streams_count++; - recompressed_png_count++; + // end uncompressed data + compressed_data_found = true; + end_uncompressed_data(); - if (DEBUG_MODE) { - cout << "Best match with level combination " << best_compression << best_mem_level << ": " << best_identical_bytes << " bytes, decompressed to " << best_identical_bytes_decomp << " bytes" << endl; - } + debug_pos(); - if (!(comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9] == -1)) { - if (fast_mode) { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - for (int i = 0; i < 81; i++) { - if (i != ((best_compression - 1) + (best_mem_level - 1) * 9)) { - comp_mem_level_count[i] = -1; - } - } - anything_was_used = true; - } else { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - anything_was_used = true; - } - } + // write compressed data header (PNG) + fout_fput_deflate_hdr(D_PNG, 0, rdres, zlib_header, 2, true); - // end uncompressed data - - compressed_data_found = true; - end_uncompressed_data(); - - // write compressed data header (PNG) - - if (best_penalty_bytes_len == 0) { - fout_fputc(1 + (best_compression << 2)); - } else { - fout_fputc(1 + 2 + (best_compression << 2)); - } - fout_fputc(3); // PNG - fout_fputc((((-windowbits) - 8) << 4) + best_mem_level); - - // store zLib header, but increased by 1 to prevent finding it - // again in the next pass - own_fwrite(zlib_header, 1, 1, fout); - unsigned char incchar = *(zlib_header + 1) + 1; - fout_fputc(incchar); - - // store penalty bytes, if any - if (best_penalty_bytes_len != 0) { - if (DEBUG_MODE) { - printf("Penalty bytes were used: %i bytes\n", best_penalty_bytes_len); - } - fout_fput32(best_penalty_bytes_len); - for (int pbc = 0; pbc < best_penalty_bytes_len; pbc++) { - fout_fputc(best_penalty_bytes[pbc]); - } - } - - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); - - // write decompressed data - - write_decompressed_data_io_buf(best_identical_bytes_decomp, in_memory); - - // start new uncompressed data - - // set input file pointer after recompressed data - input_file_pos += best_identical_bytes - 1; - cb += best_identical_bytes - 1; - - } else { - if (intense_mode_is_active()) intense_ignore_offsets->insert(input_file_pos - 2); - if (brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); - if (DEBUG_MODE) { - printf("No matches\n"); - } - } - - } + // write reconstruction and decompressed data + fout_fput_recon_data(rdres); + fout_fput_uncompressed(rdres); + debug_pos(); + // set input file pointer after recompressed data + input_file_pos += rdres.compressed_stream_size - 1; + cb += rdres.compressed_stream_size - 1; + } else { + if (intense_mode_is_active()) intense_ignore_offsets->insert(input_file_pos - 2); + if (brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); + if (DEBUG_MODE) { + printf("No matches\n"); + } + } + } } -void try_decompression_png_multi(int windowbits) { +void try_decompression_png_multi(FILE* fpng, int windowbits) { init_decompression_variables(); - // try to decompress at current position - long long compressed_stream_size = -1; - bool in_memory; - retval = try_to_decompress(fpng, windowbits, compressed_stream_size, in_memory); + // try to decompress at current position + recompress_deflate_result rdres = try_recompression_deflate(fpng); - if (retval > 0) { // seems to be a zLib-Stream + if (rdres.uncompressed_stream_size > 0) { // seems to be a zLib-Stream - decompressed_streams_count++; - decompressed_png_multi_count++; + decompressed_streams_count++; + decompressed_png_multi_count++; - if (DEBUG_MODE) { - print_debug_percent(); - cout << "Possible zLib-Stream in multiPNG found at position " << saved_input_file_pos << ", windowbits = " << -windowbits << endl; - cout << "Compressed size: " << compressed_stream_size << endl; - cout << "Can be decompressed to " << retval << " bytes" << endl; - } + debug_deflate_detected(rdres, "in multiPNG"); - for (int index = MTF.First(); index>=0; index=MTF.Next()){ - if (comp_mem_level_count[index] == -1) continue; - int comp_level = (index % 9) + 1; - int mem_level = (index / 9) + 1; + if (rdres.accepted) { + recompressed_streams_count++; + recompressed_png_multi_count++; - try_recompress(fpng, comp_level, mem_level, windowbits, compressed_stream_size, retval, in_memory); + non_zlib_was_used = true; - if (final_compression_found){ MTF.Update(); break; } - } + debug_sums(rdres); - if ((best_identical_bytes > min_ident_size) && (best_identical_bytes < best_identical_bytes_decomp)) { - recompressed_streams_count++; - recompressed_png_multi_count++; + // end uncompressed data + compressed_data_found = true; + end_uncompressed_data(); - if (DEBUG_MODE) { - cout << "Best match with level combination " << best_compression << best_mem_level << ": " << best_identical_bytes << " bytes, decompressed to " << best_identical_bytes_decomp << " bytes" << endl; - } + debug_pos(); - if (!(comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9] == -1)) { - if (fast_mode) { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - for (int i = 0; i < 81; i++) { - if (i != ((best_compression - 1) + (best_mem_level - 1) * 9)) { - comp_mem_level_count[i] = -1; - } - } - anything_was_used = true; - } else { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - anything_was_used = true; - } - } + // write compressed data header (PNG) + fout_fput_deflate_hdr(D_MULTIPNG, 0, rdres, zlib_header, 2, true); - // end uncompressed data + // simulate IDAT write to get IDAT pairs count + int i = 1; + int idat_pos = idat_lengths[0] - 2; + unsigned int idat_pairs_written_count = 0; + if (idat_pos <= rdres.compressed_stream_size) { + do { + idat_pairs_written_count++; - compressed_data_found = true; - end_uncompressed_data(); + idat_pos += idat_lengths[i]; + if (idat_pos > rdres.compressed_stream_size) break; - // write compressed data header (PNG) + i++; + } while (i < idat_count); + } + // store IDAT pairs count + fout_fput_vlint(idat_pairs_written_count); - if (best_penalty_bytes_len == 0) { - fout_fputc(1 + (best_compression << 2)); - } else { - fout_fputc(1 + 2 + (best_compression << 2)); - } - fout_fputc(4); // PNG multi - fout_fputc((((-windowbits) - 8) << 4) + best_mem_level); + // store IDAT CRCs and lengths + fout_fput_vlint(idat_lengths[0]); - // store zLib header, but increased by 1 to prevent finding it - // again in the next pass - own_fwrite(zlib_header, 1, 1, fout); - unsigned char incchar = *(zlib_header + 1) + 1; - fout_fputc(incchar); + // store IDAT CRCs and lengths + i = 1; + idat_pos = idat_lengths[0] - 2; + idat_pairs_written_count = 0; + if (idat_pos <= rdres.compressed_stream_size) { + do { + fout_fput32(idat_crcs[i]); + fout_fput_vlint(idat_lengths[i]); - // simulate IDAT write to get IDAT pairs count - int i = 1; - int idat_pos = idat_lengths[0] - 2; - unsigned int idat_pairs_written_count = 0; - if (idat_pos <= best_identical_bytes) { - do { - idat_pairs_written_count++; + idat_pairs_written_count++; - idat_pos += idat_lengths[i]; - if (idat_pos > best_identical_bytes) break; + idat_pos += idat_lengths[i]; + if (idat_pos > rdres.compressed_stream_size) break; - i++; - } while (i < idat_count); - } - // store IDAT pairs count - fout_fput16(idat_pairs_written_count); + i++; + } while (i < idat_count); + } - // store IDAT CRCs and lengths - fout_fput32(idat_lengths[0]); + // write reconstruction and decompressed data + fout_fput_recon_data(rdres); + fout_fput_uncompressed(rdres); - // store IDAT CRCs and lengths - i = 1; - idat_pos = idat_lengths[0] - 2; - idat_pairs_written_count = 0; - if (idat_pos <= best_identical_bytes) { - do { - fout_fput32(idat_crcs[i]); - fout_fput32(idat_lengths[i]); - - idat_pairs_written_count++; - - idat_pos += idat_lengths[i]; - if (idat_pos > best_identical_bytes) break; - - i++; - } while (i < idat_count); - } - - // store penalty bytes, if any - if (best_penalty_bytes_len != 0) { - if (DEBUG_MODE) { - printf("Penalty bytes were used: %i bytes\n", best_penalty_bytes_len); - } - fout_fput32(best_penalty_bytes_len); - for (int pbc = 0; pbc < best_penalty_bytes_len; pbc++) { - fout_fputc(best_penalty_bytes[pbc]); - } - } - - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); - - // write decompressed data - - write_decompressed_data_io_buf(best_identical_bytes_decomp, in_memory); - - // start new uncompressed data - - // set input file pointer after recompressed data - input_file_pos += best_identical_bytes - 1; - cb += best_identical_bytes - 1; - // now add IDAT chunk overhead - input_file_pos += (idat_pairs_written_count * 12); - cb += (idat_pairs_written_count * 12); - - } else { - if (intense_mode_is_active()) intense_ignore_offsets->insert(input_file_pos - 2); - if (brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); - if (DEBUG_MODE) { - printf("No matches\n"); - } - } - - } + debug_pos(); + // set input file pointer after recompressed data + input_file_pos += rdres.compressed_stream_size - 1; + cb += rdres.compressed_stream_size - 1; + // now add IDAT chunk overhead + input_file_pos += (idat_pairs_written_count * 12); + cb += (idat_pairs_written_count * 12); + } else { + if (intense_mode_is_active()) intense_ignore_offsets->insert(input_file_pos - 2); + if (brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); + if (DEBUG_MODE) { + printf("No matches\n"); + } + } + } } // GIF functions @@ -7593,10 +6415,10 @@ void try_decompression_gif(unsigned char version[5]) { if (recompress_success_needed) add_bits += 128; fout_fputc(1 + add_bits); - fout_fputc(5); // GIF + fout_fputc(D_GIF); // GIF // store diff bytes - fout_fput32(gDiff.GIFDiffIndex); + fout_fput_vlint(gDiff.GIFDiffIndex); if(DEBUG_MODE) { if (gDiff.GIFDiffIndex > 0) printf("Diff bytes were used: %i bytes\n", gDiff.GIFDiffIndex); @@ -7611,15 +6433,15 @@ void try_decompression_gif(unsigned char version[5]) { printf("Penalty bytes were used: %i bytes\n", best_penalty_bytes_len); } - fout_fput32(best_penalty_bytes_len); + fout_fput_vlint(best_penalty_bytes_len); for (int pbc = 0; pbc < best_penalty_bytes_len; pbc++) { fout_fputc(best_penalty_bytes[pbc]); } } - fout_fput64(best_identical_bytes); - fout_fput64(decomp_length); + fout_fput_vlint(best_identical_bytes); + fout_fput_vlint(decomp_length); // write decompressed data write_decompressed_data(decomp_length); @@ -7831,10 +6653,10 @@ void try_decompression_jpg (long long jpg_length, bool progressive_jpg) { } else { fout_fputc(1); // no penalty bytes } - fout_fputc(6); // JPG + fout_fputc(D_JPG); // JPG - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); + fout_fput_vlint(best_identical_bytes); + fout_fput_vlint(best_identical_bytes_decomp); // write compressed JPG if (in_memory) { @@ -7977,10 +6799,10 @@ void try_decompression_mp3 (long long mp3_length) { // write compressed data header (MP3) fout_fputc(1); // no penalty bytes - fout_fputc(10); // MP3 + fout_fputc(D_MP3); // MP3 - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); + fout_fput_vlint(best_identical_bytes); + fout_fput_vlint(best_identical_bytes_decomp); // write compressed MP3 if (in_memory) { @@ -8075,387 +6897,21 @@ inline unsigned short mp3_calc_layer3_crc(unsigned char header2, unsigned char h } void try_decompression_zlib(int windowbits) { - init_decompression_variables(); - - // try to decompress at current position - long long compressed_stream_size = -1; - bool in_memory; - retval = try_to_decompress(fin, windowbits, compressed_stream_size, in_memory); - - if (retval > 0) { // seems to be a zLib-Stream - - decompressed_streams_count++; - decompressed_zlib_count++; - - if (DEBUG_MODE) { - print_debug_percent(); - cout << "Possible zLib-Stream (intense mode) found at position " << saved_input_file_pos << ", windowbits = " << -windowbits << endl; - cout << "Compressed size: " << compressed_stream_size << endl; - cout << "Can be decompressed to " << retval << " bytes" << endl; - } - - for (int index = MTF.First(); index>=0; index=MTF.Next()){ - if (comp_mem_level_count[index] == -1) continue; - int comp_level = (index % 9) + 1; - int mem_level = (index / 9) + 1; - - try_recompress(fin, comp_level, mem_level, windowbits, compressed_stream_size, retval, in_memory); - - if (final_compression_found){ MTF.Update(); break; } - } - - if ((best_identical_bytes > min_ident_size_intense_brute_mode) && (best_identical_bytes < best_identical_bytes_decomp)) { - recompressed_streams_count++; - recompressed_zlib_count++; - - if (DEBUG_MODE) { - cout << "Best match with level combination " << best_compression << best_mem_level << ": " << best_identical_bytes << " bytes, decompressed to " << best_identical_bytes_decomp << " bytes" << endl; - } - - if (!(comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9] == -1)) { - if (fast_mode) { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - for (int i = 0; i < 81; i++) { - if (i != ((best_compression - 1) + (best_mem_level - 1) * 9)) { - comp_mem_level_count[i] = -1; - } - } - anything_was_used = true; - } else { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - anything_was_used = true; - } - } - - // end uncompressed data - - compressed_data_found = true; - end_uncompressed_data(); - - // check recursion - write_ftempout_if_not_present(best_identical_bytes_decomp, in_memory); - recursion_result r = recursion_compress(best_identical_bytes, best_identical_bytes_decomp); - - // write compressed data header (zLib) - - int header_byte = 1 + (best_compression << 2); - if (best_penalty_bytes_len != 0) { - header_byte += 2; - } - if (r.success) { - header_byte += 128; - } - fout_fputc(header_byte); - fout_fputc(255); // raw zLib - fout_fputc((((-windowbits) - 8) << 4) + best_mem_level); - - // store zLib header, but increased by 1 to prevent finding it - // again in the next pass - own_fwrite(in_buf + cb, 1, 1, fout); - unsigned char incchar = *(in_buf + cb + 1) + 1; - fout_fputc(incchar); - - // store penalty bytes, if any - if (best_penalty_bytes_len != 0) { - if (DEBUG_MODE) { - printf("Penalty bytes were used: %i bytes\n", best_penalty_bytes_len); - } - fout_fput32(best_penalty_bytes_len); - for (int pbc = 0; pbc < best_penalty_bytes_len; pbc++) { - fout_fputc(best_penalty_bytes[pbc]); - } - } - - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); - - if (r.success) { - fout_fput64(r.file_length); - } - - // write decompressed data - if (r.success) { - write_decompressed_data(r.file_length, r.file_name); - remove(r.file_name); - delete[] r.file_name; - } else { - write_decompressed_data_io_buf(best_identical_bytes_decomp, in_memory); - } - - // start new uncompressed data - - // set input file pointer after recompressed data - input_file_pos += best_identical_bytes - 1; - cb += best_identical_bytes - 1; - - } else { - if (brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); - if (DEBUG_MODE) { - printf("No matches\n"); - } - } - - } - - + try_decompression_deflate_type(decompressed_zlib_count, recompressed_zlib_count, + D_RAW, in_buf + cb, 2, true, + "(intense mode)"); } void try_decompression_brute() { - init_decompression_variables(); - - int windowbits; - - // try to decompress at current position - long long compressed_stream_size = -1; - bool in_memory; - retval = try_to_decompress(fin, -15, compressed_stream_size, in_memory); - - if (retval > 0) { // seems to be a zLib-Stream - - decompressed_streams_count++; - decompressed_brute_count++; - - if (DEBUG_MODE) { - print_debug_percent(); - cout << "Possible zLib-Stream (brute mode) found at position " << saved_input_file_pos << endl; - cout << "Compressed size: " << compressed_stream_size << endl; - cout << "Can be decompressed to " << retval << " bytes" << endl; - } - - for (windowbits = -15; windowbits < -7; windowbits++) { - for (int index = MTF.First(); index>=0; index=MTF.Next()){ - if (comp_mem_level_count[index] == -1) continue; - int comp_level = (index % 9) + 1; - int mem_level = (index / 9) + 1; - - try_recompress(fin, comp_level, mem_level, windowbits, compressed_stream_size, retval, in_memory); - - if (final_compression_found){ MTF.Update(); break; } - } - if (final_compression_found) break; - } - - if ((best_identical_bytes > min_ident_size_intense_brute_mode) && (best_identical_bytes < best_identical_bytes_decomp)) { - recompressed_streams_count++; - recompressed_brute_count++; - - windowbits = best_windowbits; - if (DEBUG_MODE) { - cout << "Best match with level combination " << best_compression << best_mem_level << ", windowbits = " << -windowbits << ": " << best_identical_bytes << " bytes, decompressed to " << best_identical_bytes_decomp << " bytes" << endl; - } - - if (!(comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9] == -1)) { - if (fast_mode) { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - for (int i = 0; i < 81; i++) { - if (i != ((best_compression - 1) + (best_mem_level - 1) * 9)) { - comp_mem_level_count[i] = -1; - } - } - anything_was_used = true; - } else { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - anything_was_used = true; - } - } - - // end uncompressed data - - compressed_data_found = true; - end_uncompressed_data(); - - // check recursion - write_ftempout_if_not_present(best_identical_bytes_decomp, in_memory); - recursion_result r = recursion_compress(best_identical_bytes, best_identical_bytes_decomp); - - // write compressed data header (brute) - - int header_byte = 1 + (best_compression << 2); - if (best_penalty_bytes_len != 0) { - header_byte += 2; - } - if (r.success) { - header_byte += 128; - } - fout_fputc(header_byte); - fout_fputc(254); // brute - fout_fputc((((-windowbits) - 8) << 4) + best_mem_level); - - // store penalty bytes, if any - if (best_penalty_bytes_len != 0) { - if (DEBUG_MODE) { - printf("Penalty bytes were used: %i bytes\n", best_penalty_bytes_len); - } - fout_fput32(best_penalty_bytes_len); - for (int pbc = 0; pbc < best_penalty_bytes_len; pbc++) { - fout_fputc(best_penalty_bytes[pbc]); - } - } - - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); - - if (r.success) { - fout_fput64(r.file_length); - } - - // write decompressed data - if (r.success) { - write_decompressed_data(r.file_length, r.file_name); - remove(r.file_name); - delete[] r.file_name; - } else { - write_decompressed_data_io_buf(best_identical_bytes_decomp, in_memory); - } - - // set input file pointer after recompressed data - input_file_pos += best_identical_bytes - 1; - cb += best_identical_bytes - 1; - - } else { - if (DEBUG_MODE) { - printf("No matches\n"); - } - } - - } - - + try_decompression_deflate_type(decompressed_brute_count, recompressed_brute_count, + D_BRUTE, in_buf + cb, 0, false, + "(brute mode)"); } void try_decompression_swf(int windowbits) { - init_decompression_variables(); - - // try to decompress at current position - long long compressed_stream_size = -1; - bool in_memory; - retval = try_to_decompress(fin, windowbits, compressed_stream_size, in_memory); - - if (retval > 0) { // seems to be a zLib-Stream - - decompressed_streams_count++; - decompressed_swf_count++; - - if (DEBUG_MODE) { - print_debug_percent(); - cout << "Possible zLib-Stream in SWF found at position " << saved_input_file_pos << ", windowbits = " << -windowbits << endl; - cout << "Compressed size: " << compressed_stream_size << endl; - cout << "Can be decompressed to " << retval << " bytes" << endl; - } - - for (int index = MTF.First(); index>=0; index=MTF.Next()){ - if (comp_mem_level_count[index] == -1) continue; - int comp_level = (index % 9) + 1; - int mem_level = (index / 9) + 1; - - try_recompress(fin, comp_level, mem_level, windowbits, compressed_stream_size, retval, in_memory); - - if (final_compression_found){ MTF.Update(); break; } - } - - if ((best_identical_bytes > min_ident_size) && (best_identical_bytes < best_identical_bytes_decomp)) { - recompressed_streams_count++; - recompressed_swf_count++; - - if (DEBUG_MODE) { - cout << "Best match with level combination " << best_compression << best_mem_level << ": " << best_identical_bytes << " bytes, decompressed to " << best_identical_bytes_decomp << " bytes" << endl; - } - - if (!(comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9] == -1)) { - if (fast_mode) { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - for (int i = 0; i < 81; i++) { - if (i != ((best_compression - 1) + (best_mem_level - 1) * 9)) { - comp_mem_level_count[i] = -1; - } - } - anything_was_used = true; - } else { - comp_mem_level_count[(best_compression - 1) + (best_mem_level - 1) * 9]++; - zlib_level_was_used[(best_compression - 1) + (best_mem_level - 1) * 9] = true; - anything_was_used = true; - } - } - - // end uncompressed data - - compressed_data_found = true; - end_uncompressed_data(); - - // check recursion - write_ftempout_if_not_present(best_identical_bytes_decomp, in_memory); - recursion_result r = recursion_compress(best_identical_bytes, best_identical_bytes_decomp); - - // write compressed data header (SWF) - - int header_byte = 1 + (best_compression << 2); - if (best_penalty_bytes_len != 0) { - header_byte += 2; - } - if (r.success) { - header_byte += 128; - } - fout_fputc(header_byte); - fout_fputc(7); // SWF - fout_fputc((((-windowbits) - 8) << 4) + best_mem_level); - - // store version and length from SWF header - own_fwrite(in_buf + cb + 3, 1, 5, fout); - - // store zLib header, but increased by 1 to prevent finding it - // again in the next pass - own_fwrite(in_buf + cb + 8, 1, 1, fout); - unsigned char incchar = *(in_buf + cb + 9) + 1; - fout_fputc(incchar); - - // store penalty bytes, if any - if (best_penalty_bytes_len != 0) { - if (DEBUG_MODE) { - printf("Penalty bytes were used: %i bytes\n", best_penalty_bytes_len); - } - fout_fput32(best_penalty_bytes_len); - for (int pbc = 0; pbc < best_penalty_bytes_len; pbc++) { - fout_fputc(best_penalty_bytes[pbc]); - } - } - - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); - - if (r.success) { - fout_fput64(r.file_length); - } - - // write decompressed data - if (r.success) { - write_decompressed_data(r.file_length, r.file_name); - remove(r.file_name); - delete[] r.file_name; - } else { - write_decompressed_data_io_buf(best_identical_bytes_decomp, in_memory); - } - - // start new uncompressed data - - // set input file pointer after recompressed data - input_file_pos += best_identical_bytes - 1; - cb += best_identical_bytes - 1; - - } else { - if (intense_mode_is_active()) intense_ignore_offsets->insert(input_file_pos - 2); - if (brute_mode_is_active()) brute_ignore_offsets->insert(input_file_pos); - if (DEBUG_MODE) { - printf("No matches\n"); - } - } - - } - + try_decompression_deflate_type(decompressed_swf_count, recompressed_swf_count, + D_SWF, in_buf + cb + 3, 7, true, + "in SWF"); } void try_decompression_bzip2(int compression_level) { @@ -8511,7 +6967,7 @@ void try_decompression_bzip2(int compression_level) { header_byte += 128; } fout_fputc(header_byte); - fout_fputc(9); // Base64 + fout_fputc(D_BZIP2); // Base64 fout_fputc(compression_level); // store penalty bytes, if any @@ -8519,17 +6975,17 @@ void try_decompression_bzip2(int compression_level) { if (DEBUG_MODE) { printf("Penalty bytes were used: %i bytes\n", best_penalty_bytes_len); } - fout_fput32(best_penalty_bytes_len); + fout_fput_vlint(best_penalty_bytes_len); for (int pbc = 0; pbc < best_penalty_bytes_len; pbc++) { fout_fputc(best_penalty_bytes[pbc]); } } - fout_fput64(best_identical_bytes); - fout_fput64(best_identical_bytes_decomp); + fout_fput_vlint(best_identical_bytes); + fout_fput_vlint(best_identical_bytes_decomp); if (r.success) { - fout_fput64(r.file_length); + fout_fput_vlint(r.file_length); } // write decompressed data @@ -8835,15 +7291,15 @@ void try_decompression_base64(int base64_header_length) { header_byte += 128; } fout_fputc(header_byte); - fout_fputc(8); // Base64 + fout_fputc(D_BASE64); // Base64 - fout_fput16(base64_header_length); + fout_fput_vlint(base64_header_length); // write "header", but change first char to prevent re-detection fout_fputc(in_buf[cb] - 1); own_fwrite(in_buf + cb + 1, 1, base64_header_length - 1, fout); - fout_fput16(line_count); + fout_fput_vlint(line_count); if (line_case == 2) { for (i = 0; i < line_count; i++) { fout_fputc(base64_line_len[i]); @@ -8855,11 +7311,11 @@ void try_decompression_base64(int base64_header_length) { delete[] base64_line_len; - fout_fput64(identical_bytes); - fout_fput64(identical_bytes_decomp); + fout_fput_vlint(identical_bytes); + fout_fput_vlint(identical_bytes_decomp); if (r.success) { - fout_fput64(r.file_length); + fout_fput_vlint(r.file_length); } // write decompressed data @@ -9316,6 +7772,11 @@ recursion_result recursion_compress(long long compressed_bytes, long long decomp return tmp_r; } +recursion_result recursion_write_file_and_compress(const recompress_deflate_result& rdres) { + write_ftempout_if_not_present(rdres.uncompressed_stream_size, rdres.uncompressed_in_memory); + recursion_result r = recursion_compress(rdres.compressed_stream_size, rdres.uncompressed_stream_size); + return r; +} recursion_result recursion_decompress(long long recursion_data_length) { FILE* recursion_fin; @@ -9408,17 +7869,6 @@ void fout_fputc(char c) { } } -void fout_fput16(int v) { - fout_fputc((v >> 8) % 256); - fout_fputc(v % 256); -} - -void fout_fput24(int v) { - fout_fputc((v >> 16) % 256); - fout_fputc((v >> 8) % 256); - fout_fputc(v % 256); -} - void fout_fput32_little_endian(int v) { fout_fputc(v % 256); fout_fputc((v >> 8) % 256); @@ -9440,26 +7890,116 @@ void fout_fput32(unsigned int v) { fout_fputc(v % 256); } -void fout_fput64(long long v) { - fout_fputc((v >> 56) % 256); - fout_fputc((v >> 48) % 256); - fout_fputc((v >> 40) % 256); - fout_fputc((v >> 32) % 256); - fout_fputc((v >> 24) % 256); - fout_fputc((v >> 16) % 256); - fout_fputc((v >> 8) % 256); - fout_fputc(v % 256); +void fout_fput_vlint(unsigned long long v) { + while (v >= 128) { + fout_fputc((v & 127) + 128); + v = (v >> 7) - 1; + } + fout_fputc(v); } +void fout_fput_deflate_hdr(const unsigned char type, const unsigned char flags, + const recompress_deflate_result& rdres, + const unsigned char* hdr, const unsigned hdr_length, + const bool inc_last_hdr_byte) { + fout_fputc(1 + (rdres.zlib_perfect ? rdres.zlib_comp_level << 2 : 2) + flags); + fout_fputc(type); // PDF/PNG/... + if (rdres.zlib_perfect) { + fout_fputc(((rdres.zlib_window_bits - 8) << 4) + rdres.zlib_mem_level); + } + fout_fput_vlint(hdr_length); + if (!inc_last_hdr_byte) { + own_fwrite(hdr, 1, hdr_length, fout); + } else { + own_fwrite(hdr, 1, hdr_length - 1, fout); + fout_fputc(hdr[hdr_length - 1] + 1); + } +} +void fin_fget_deflate_hdr(recompress_deflate_result& rdres, const unsigned char flags, + unsigned char* hdr_data, unsigned& hdr_length, + const bool inc_last_hdr_byte) { + rdres.zlib_perfect = (flags & 2) == 0; + if (rdres.zlib_perfect) { + unsigned char zlib_params = fin_fgetc(); + rdres.zlib_comp_level = (flags & 0x3c) >> 2; + rdres.zlib_mem_level = zlib_params & 0x0f; + rdres.zlib_window_bits = ((zlib_params >> 4) & 0x7) + 8; + } + hdr_length = fin_fget_vlint(); + if (!inc_last_hdr_byte) { + own_fread(hdr_data, 1, hdr_length, fin); + } else { + own_fread(hdr_data, 1, hdr_length - 1, fin); + hdr_data[hdr_length - 1] = fin_fgetc() - 1; + } + own_fwrite(hdr_data, 1, hdr_length, fout); +} +void fout_fput_recon_data(const recompress_deflate_result& rdres) { + if (!rdres.zlib_perfect) { + fout_fput_vlint(rdres.recon_data.size()); + own_fwrite(rdres.recon_data.data(), 1, rdres.recon_data.size(), fout); + } -void fout_fput64(unsigned long long v) { - fout_fputc((v >> 56) % 256); - fout_fputc((v >> 48) % 256); - fout_fputc((v >> 40) % 256); - fout_fputc((v >> 32) % 256); - fout_fputc((v >> 24) % 256); - fout_fputc((v >> 16) % 256); - fout_fputc((v >> 8) % 256); - fout_fputc(v % 256); + fout_fput_vlint(rdres.compressed_stream_size); + fout_fput_vlint(rdres.uncompressed_stream_size); +} +void fin_fget_recon_data(recompress_deflate_result& rdres) { + if (!rdres.zlib_perfect) { + size_t sz = fin_fget_vlint(); + rdres.recon_data.resize(sz); + own_fread(rdres.recon_data.data(), 1, rdres.recon_data.size(), fin); + } + + rdres.compressed_stream_size = fin_fget_vlint(); + rdres.uncompressed_stream_size = fin_fget_vlint(); +} +void fout_fput_uncompressed(const recompress_deflate_result& rdres) { + write_decompressed_data_io_buf(rdres.uncompressed_stream_size, rdres.uncompressed_in_memory); +} +void fin_fget_uncompressed(const recompress_deflate_result&) { +} +void fout_fput_deflate_rec(const unsigned char type, + const recompress_deflate_result& rdres, + const unsigned char* hdr, const unsigned hdr_length, const bool inc_last, + const recursion_result& recres) { + fout_fput_deflate_hdr(type, recres.success ? 128 : 0, rdres, hdr, hdr_length, inc_last); + fout_fput_recon_data(rdres); + + // write decompressed data + if (recres.success) { + fout_fput_vlint(recres.file_length); + write_decompressed_data(recres.file_length, recres.file_name); + remove(recres.file_name); + delete[] recres.file_name; + } else { + fout_fput_uncompressed(rdres); + } +} +bool fin_fget_deflate_rec(recompress_deflate_result& rdres, const unsigned char flags, + unsigned char* hdr, unsigned& hdr_length, const bool inc_last, + int64_t& recursion_length) { + fin_fget_deflate_hdr(rdres, flags, hdr, hdr_length, inc_last); + fin_fget_recon_data(rdres); + + debug_sums(rdres); + + // write decompressed data + if (flags & 128) { + recursion_length = fin_fget_vlint(); + recursion_result r = recursion_decompress(recursion_length); + debug_pos(); + bool result = try_reconstructing_deflate(r.frecurse, fout, rdres); + debug_pos(); + safe_fclose(&r.frecurse); + remove(r.file_name); + delete[] r.file_name; + return result; + } else { + recursion_length = 0; + debug_pos(); + bool result = try_reconstructing_deflate(fin, fout, rdres); + debug_pos(); + return result; + } } unsigned char fin_fgetc() { @@ -9473,19 +8013,33 @@ unsigned char fin_fgetc() { return temp_buf[0]; } } - -long long fin_fget64() { - long long result = ((long long)fin_fgetc() << 56); - result += ((long long)fin_fgetc() << 48); - result += ((long long)fin_fgetc() << 40); - result += ((long long)fin_fgetc() << 32); - result += ((long long)fin_fgetc() << 24); - result += ((long long)fin_fgetc() << 16); - result += ((long long)fin_fgetc() << 8); - result += (long long)fin_fgetc(); - +int32_t fin_fget32_little_endian() { + int32_t result = 0; + result += ((long)fin_fgetc() << 0); + result += ((long)fin_fgetc() << 8); + result += ((long)fin_fgetc() << 16); + result += ((long)fin_fgetc() << 24); return result; } +int32_t fin_fget32() { + int32_t result = 0; + result += ((long)fin_fgetc() << 24); + result += ((long)fin_fgetc() << 16); + result += ((long)fin_fgetc() << 8); + result += (long)fin_fgetc(); + return result; +} +long long fin_fget_vlint() { + unsigned char c; + long long v = 0, o = 0, s = 0; + while ((c = fin_fgetc()) >= 128) { + v += (((long long)(c & 127)) << s); + s += 7; + o = (o + 1) << 7; + } + return v + o + (((long long)c) << s); +} + void init_compress_otf() { if (comp_decomp_state == P_CONVERT) compression_otf_method = conversion_to_method; diff --git a/precomp.h b/precomp.h index 5062fdd..e52626b 100644 --- a/precomp.h +++ b/precomp.h @@ -3,7 +3,6 @@ long long def_compare(FILE *compfile, int level, int windowbits, int memlevel, l long long def_compare_bzip2(FILE *source, FILE *compfile, int level, long long& decompressed_bytes_used); int def_part(FILE *source, FILE *dest, int level, int windowbits, int memlevel, long long stream_size_in, long long stream_size_out); int def_part_skip(FILE *source, FILE *dest, int level, int windowbits, int memlevel, long long stream_size_in, long long stream_size_out, int bmp_width); -int inf(FILE *source, int windowbits, long long& compressed_stream_size, long long& decompressed_stream_size, bool& in_memory); void zerr(int ret); #ifndef PRECOMPDLL #ifndef COMFORT @@ -32,7 +31,7 @@ void try_decompression_pdf(int windowbits, int pdf_header_length, int img_width, void try_decompression_zip(int zip_header_length); void try_decompression_gzip(int gzip_header_length); void try_decompression_png(int windowbits); -void try_decompression_png_multi(int windowbits); +void try_decompression_png_multi(FILE* fpng, int windowbits); void try_decompression_gif(unsigned char version[5]); void try_decompression_jpg(long long jpg_length, bool progressive_jpg); void try_decompression_mp3(long long mp3_length); @@ -129,24 +128,33 @@ public: List[Root=Index].Previous = -1; } }; +struct recompress_deflate_result; void write_ftempout_if_not_present(long long byte_count, bool in_memory, bool leave_open = false); recursion_result recursion_compress(long long compressed_bytes, long long decompressed_bytes); recursion_result recursion_decompress(long long recursion_data_length); +recursion_result recursion_write_file_and_compress(const recompress_deflate_result&); // compression-on-the-fly enum {OTF_NONE = 0, OTF_BZIP2 = 1, OTF_XZ_MT = 2}; // uncompressed, bzip2, lzma2 multithreaded void own_fputc(char c, FILE* f); unsigned char fin_fgetc(); -long long fin_fget64(); +int32_t fin_fget32_little_endian(); +int32_t fin_fget32(); +long long fin_fget_vlint(); +void fin_fget_deflate_hdr(recompress_deflate_result&, const unsigned char flags, unsigned char* hdr_data, unsigned& hdr_length, const bool inc_last); +void fin_fget_recon_data(recompress_deflate_result&); +bool fin_fget_deflate_rec(recompress_deflate_result&, const unsigned char flags, unsigned char* hdr, unsigned& hdr_length, const bool inc_last, int64_t& rec_length); +void fin_fget_uncompressed(const recompress_deflate_result&); void fout_fputc(char c); -void fout_fput16(int v); -void fout_fput24(int v); void fout_fput32_little_endian(int v); void fout_fput32(int v); void fout_fput32(unsigned int v); -void fout_fput64(long long v); -void fout_fput64(unsigned long long v); +void fout_fput_vlint(unsigned long long v); +void fout_fput_deflate_hdr(const unsigned char type, const unsigned char flags, const recompress_deflate_result&, const unsigned char* hdr_data, const unsigned hdr_length, const bool inc_last); +void fout_fput_recon_data(const recompress_deflate_result&); +void fout_fput_deflate_rec(const unsigned char type, const recompress_deflate_result&, const unsigned char* hdr, const unsigned hdr_length, const bool inc_last, const recursion_result& recres); +void fout_fput_uncompressed(const recompress_deflate_result&); void init_compress_otf(); void denit_compress_otf(); void init_decompress_otf();